2025-04-04 18:32:16,113 [ 917585 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-04-04 18:32:16,114 [ 917585 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:97, check_args_and_update_paths) 2025-04-04 18:32:16,114 [ 917585 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:108, check_args_and_update_paths) 2025-04-04 18:32:16,114 [ 917585 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:110, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_e3k51h --privileged --dns-search='.' --memory=30709026816 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=2 --color=no --durations=0 test_storage_azure_blob_storage/test_cluster.py::test_select_all 'test_storage_iceberg/test.py::test_cluster_table_function[azure-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[azure-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-2]' -vvv" altinityinfra/integration-tests-runner:cd6390247eca '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: random-0.2, timeout-2.2.0, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False collecting ... collected 7 items test_storage_azure_blob_storage/test_cluster.py::test_select_all FAILED [ 14%] test_storage_iceberg/test.py::test_cluster_table_function[azure-1] FAILED [ 28%] test_storage_iceberg/test.py::test_cluster_table_function[azure-2] FAILED [ 42%] test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1] FAILED [ 57%] test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2] FAILED [ 71%] test_storage_iceberg/test.py::test_cluster_table_function[s3-1] FAILED [ 85%] test_storage_iceberg/test.py::test_cluster_table_function[s3-2] FAILED [100%] =================================== FAILURES =================================== _______________________________ test_select_all ________________________________ cluster = def test_select_all(cluster): node = cluster.instances["node_0"] port = cluster.env_variables["AZURITE_PORT"] storage_account_url = cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"] azure_query( node, f"INSERT INTO TABLE FUNCTION azureBlobStorage('{storage_account_url}', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1'," f"'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV', 'auto', 'key UInt64, data String') " f"VALUES (1, 'a'), (2, 'b')", settings={"azure_truncate_on_insert": 1}, ) print(get_azure_file_content("test_cluster_select_all.csv", port)) query_id_pure = str(uuid.uuid4()) pure_azure = azure_query( node, f"SELECT * from azureBlobStorage('{storage_account_url}', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1'," f"'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV','auto')", query_id=query_id_pure, ) print(pure_azure) query_id_distributed = str(uuid.uuid4()) distributed_azure = azure_query( node, f"SELECT * from azureBlobStorageCluster('simple_cluster', '{storage_account_url}', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1'," f"'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV'," f"'auto')", query_id=query_id_distributed, ) print(distributed_azure) query_id_distributed_alt_syntax = str(uuid.uuid4()) distributed_azure_alt_syntax = azure_query( node, f"SELECT * from azureBlobStorage('{storage_account_url}', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1'," f"'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV'," f"'auto') " f"SETTINGS object_storage_cluster='simple_cluster'", query_id=query_id_distributed_alt_syntax, ) print(distributed_azure_alt_syntax) azure_query( node, f""" DROP TABLE IF EXISTS azure_engine_table_single_node; CREATE TABLE azure_engine_table_single_node (key UInt64, data String) ENGINE=AzureBlobStorage( '{storage_account_url}', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV', 'auto' ) """, ) query_id_engine_single_node = str(uuid.uuid4()) azure_engine_single_node = azure_query( node, "SELECT * FROM azure_engine_table_single_node", query_id=query_id_engine_single_node, ) azure_query( node, f""" DROP TABLE IF EXISTS azure_engine_table_distributed; CREATE TABLE azure_engine_table_distributed (key UInt64, data String) ENGINE=AzureBlobStorage( '{storage_account_url}', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV', 'auto' ) SETTINGS object_storage_cluster='simple_cluster' """, ) query_id_engine_distributed = str(uuid.uuid4()) azure_engine_distributed = azure_query( node, "SELECT * FROM azure_engine_table_distributed", query_id=query_id_engine_distributed, ) assert TSV(pure_azure) == TSV(distributed_azure) assert TSV(pure_azure) == TSV(distributed_azure_alt_syntax) assert TSV(pure_azure) == TSV(azure_engine_single_node) > assert TSV(pure_azure) == TSV(azure_engine_distributed) E AssertionError: assert 1 a\n2 b == 1 a\n2 b\n1 a\n2 b\n1 a\n2 b E + where 1 a\n2 b = TSV('1\ta\n2\tb\n') E + and 1 a\n2 b\n1 a\n2 b\n1 a\n2 b = TSV('1\ta\n2\tb\n1\ta\n2\tb\n1\ta\n2\tb\n') test_storage_azure_blob_storage/test_cluster.py:157: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2025-04-04 18:32:20 [ 633 ] DEBUG : Command:[docker ps | wc -l] (cluster.py:122, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : No running containers (conftest.py:96, cleanup_environment) 2025-04-04 18:32:20 [ 633 ] DEBUG : Pruning Docker networks (conftest.py:98, cleanup_environment) 2025-04-04 18:32:20 [ 633 ] DEBUG : Command:[docker network prune --force] (cluster.py:122, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] (cluster.py:122, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:146, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV HOSTNAME 52698a6efbf1 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV SHLVL 0 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV HOME /root (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV OLDPWD / (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_HELPER_TAG 5dc43a6382f0 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV PYTEST_ADDOPTS -rfEps --run-id=2 --color=no --durations=0 test_storage_azure_blob_storage/test_cluster.py::test_select_all 'test_storage_iceberg/test.py::test_cluster_table_function[azure-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[azure-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-2]' -vvv (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV TZ Etc/UTC (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV DOCKER_BASE_TAG 6712d5cc610d (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 2 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : ENV PYTEST_CURRENT_TEST test_storage_azure_blob_storage/test_cluster.py::test_select_all (setup) (cluster.py:450, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:774, __init__) 2025-04-04 18:32:20 [ 633 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-04 18:32:20 [ 633 ] DEBUG : Cluster name:cluster project_name:rootteststorageazureblobstoragecluster. Added instance name:node_0 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env', '--project-name', 'rootteststorageazureblobstoragecluster', '--file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-04 18:32:20 [ 633 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-04 18:32:20 [ 633 ] DEBUG : Cluster name:cluster project_name:rootteststorageazureblobstoragecluster. Added instance name:node_1 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env', '--project-name', 'rootteststorageazureblobstoragecluster', '--file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml', '--file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-04 18:32:20 [ 633 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-04 18:32:20 [ 633 ] DEBUG : Cluster name:cluster project_name:rootteststorageazureblobstoragecluster. Added instance name:node_2 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env', '--project-name', 'rootteststorageazureblobstoragecluster', '--file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml', '--file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-04 18:32:20 [ 633 ] INFO : Running tests in /ClickHouse/tests/integration/test_storage_azure_blob_storage/test_cluster.py (cluster.py:2793, start) 2025-04-04 18:32:20 [ 633 ] DEBUG : Cluster start called. is_up=False (cluster.py:2800, start) 2025-04-04 18:32:20 [ 633 ] DEBUG : Docker networks for project rootteststorageazureblobstoragecluster are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-04 18:32:20 [ 633 ] DEBUG : Docker containers for project rootteststorageazureblobstoragecluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-04 18:32:20 [ 633 ] DEBUG : Docker volumes for project rootteststorageazureblobstoragecluster are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-04 18:32:20 [ 633 ] DEBUG : Cleanup called (cluster.py:894, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : Docker networks for project rootteststorageazureblobstoragecluster are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-04 18:32:20 [ 633 ] DEBUG : Docker containers for project rootteststorageazureblobstoragecluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-04 18:32:20 [ 633 ] DEBUG : Docker volumes for project rootteststorageazureblobstoragecluster are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-04 18:32:20 [ 633 ] DEBUG : Command:[docker container list --all --filter name='^/rootteststorageazureblobstoragecluster-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : No running containers for project: rootteststorageazureblobstoragecluster (cluster.py:922, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Images pruned (cluster.py:947, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-04 18:32:20 [ 633 ] DEBUG : Volumes pruned: 1 (cluster.py:958, cleanup) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup directory for instance: node_0 (cluster.py:2813, start) 2025-04-04 18:32:20 [ 633 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_azure_blob_storage/configs/named_collections.xml', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/configs/cluster.xml'] to /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/configs/config.d (cluster.py:4752, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/database (cluster.py:4769, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/logs (cluster.py:4780, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4864, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup directory for instance: node_1 (cluster.py:2813, start) 2025-04-04 18:32:20 [ 633 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_azure_blob_storage/configs/named_collections.xml', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/configs/cluster.xml'] to /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/configs/config.d (cluster.py:4752, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/database (cluster.py:4769, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/logs (cluster.py:4780, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4864, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup directory for instance: node_2 (cluster.py:2813, start) 2025-04-04 18:32:20 [ 633 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_azure_blob_storage/configs/named_collections.xml', '/ClickHouse/tests/integration/test_storage_azure_blob_storage/configs/cluster.xml'] to /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/configs/config.d (cluster.py:4752, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/database (cluster.py:4769, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/logs (cluster.py:4780, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4864, create_dir) 2025-04-04 18:32:20 [ 633 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'AZURITE_PORT': '30000', 'AZURITE_STORAGE_ACCOUNT_URL': 'http://azurite1:30000/devstoreaccount1', 'AZURITE_CONNECTION_STRING': 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite1:30000/devstoreaccount1;'} stored in /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env (cluster.py:97, _create_env_file) 2025-04-04 18:32:20 [ 633 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-04 18:32:20 [ 633 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-04 18:32:20 [ 633 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-04 18:32:20 [ 633 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-04 18:32:20 [ 633 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2025-04-04 18:32:20 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env --project-name rootteststorageazureblobstoragecluster --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/docker-compose.yml pull] (cluster.py:122, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: node_1 Skipped - Image is already being pulled by node_0 (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: node_2 Skipped - Image is already being pulled by node_0 (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: node_0 Pulling (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: azurite1 Pulling (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: node_0 Pulled (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: azurite1 Pulled (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] INFO : Trying to create Azurite instance by command docker compose --project-name rootteststorageazureblobstoragecluster --env-file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --verbose up -d (cluster.py:3143, start) 2025-04-04 18:33:22 [ 633 ] DEBUG : Command:[docker compose --project-name rootteststorageazureblobstoragecluster --env-file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --verbose up -d] (cluster.py:122, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:33:22Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Network rootteststorageazureblobstoragecluster_default Creating (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Network rootteststorageazureblobstoragecluster_default Created (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Volume "rootteststorageazureblobstoragecluster_data1-1" Creating (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Volume "rootteststorageazureblobstoragecluster_data1-1" Created (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:33:22Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:33:22Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:33:22 [ 633 ] INFO : Trying to connect to Azurite (cluster.py:3157, start) 2025-04-04 18:33:23 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30000/devstoreaccount1/?restype=REDACTED&comp=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '4a7c6188-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:33:23 [ 633 ] DEBUG : Starting new HTTP connection (1): 127.0.0.1:30000 (connectionpool.py:245, _new_conn) 2025-04-04 18:33:23 [ 633 ] DEBUG : http://127.0.0.1:30000 "GET /devstoreaccount1/?restype=account&comp=properties HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:33:23 [ 633 ] INFO : Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-client-request-id': '4a7c6188-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '2f46ad0f-e168-4871-b858-e905e89365fb' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:33:23 GMT' 'x-ms-sku-name': 'REDACTED' 'x-ms-account-kind': 'REDACTED' 'x-ms-is-hns-enabled': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:33:23 [ 633 ] DEBUG : {'client_request_id': '4a7c6188-1183-11f0-b12d-0242ac110002', 'request_id': '2f46ad0f-e168-4871-b858-e905e89365fb', 'version': '2025-05-05', 'date': datetime.datetime(2025, 4, 4, 18, 33, 23, tzinfo=datetime.timezone.utc), 'sku_name': 'Standard_RAGRS', 'account_kind': 'StorageV2', 'is_hns_enabled': False} (cluster.py:2665, wait_azurite_to_start) 2025-04-04 18:33:23 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30000/devstoreaccount1/?comp=REDACTED&prefix=REDACTED&include=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '4a8182b2-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:33:23 [ 633 ] DEBUG : http://127.0.0.1:30000 "GET /devstoreaccount1/?comp=list&prefix=azurite-container&include= HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:23 [ 633 ] INFO : Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-client-request-id': '4a8182b2-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '6f00955e-5e5a-4e76-ab91-e92eab50c835' 'x-ms-version': 'REDACTED' 'content-type': 'application/xml' 'Date': 'Fri, 04 Apr 2025 18:33:23 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' (_universal.py:550, on_response) 2025-04-04 18:33:23 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30000/devstoreaccount1/azurite-container?restype=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '4a82c870-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:33:23 [ 633 ] DEBUG : http://127.0.0.1:30000 "GET /devstoreaccount1/azurite-container?restype=container HTTP/1.1" 404 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:23 [ 633 ] INFO : Response status: 404 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-error-code': 'ContainerNotFound' 'x-ms-request-id': 'aec876d1-33b1-4fe6-b0aa-2f04888de43e' 'content-type': 'application/xml' 'Date': 'Fri, 04 Apr 2025 18:33:23 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' (_universal.py:550, on_response) 2025-04-04 18:33:23 [ 633 ] DEBUG : azurite container 'azurite-container' doesn't exist, creating it (cluster.py:2687, wait_azurite_to_start) 2025-04-04 18:33:23 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30000/devstoreaccount1/azurite-container?restype=REDACTED' Request method: 'PUT' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '4a83ae2a-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:33:23 [ 633 ] DEBUG : http://127.0.0.1:30000 "PUT /devstoreaccount1/azurite-container?restype=container HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:33:23 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x21E4578E1CC10A0"' 'last-modified': 'Fri, 04 Apr 2025 18:33:23 GMT' 'x-ms-client-request-id': '4a83ae2a-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'bcabee62-13d8-4f06-8803-1f14de2dec93' 'x-ms-version': 'REDACTED' 'Date': 'Fri, 04 Apr 2025 18:33:23 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:33:23 [ 633 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env --project-name rootteststorageazureblobstoragecluster --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/docker-compose.yml up -d --no-recreate') (cluster.py:3200, start) 2025-04-04 18:33:23 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env --project-name rootteststorageazureblobstoragecluster --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/docker-compose.yml up -d --no-recreate] (cluster.py:122, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Running (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:33:24 [ 633 ] DEBUG : ClickHouse instance created (cluster.py:3208, start) 2025-04-04 18:33:24 [ 633 ] DEBUG : get_instance_ip instance_name=node_0 (cluster.py:2082, get_instance_ip) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : get_instance_ip instance_name=node_0 (cluster.py:2092, get_instance_global_ipv6) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : Waiting for ClickHouse start in node_0, ip: 172.16.1.4... (cluster.py:3216, start) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/396ec9e0c6096a383b6825a959cf12e798c0e03938d8d1145f3ee83eb3322f10/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/396ec9e0c6096a383b6825a959cf12e798c0e03938d8d1145f3ee83eb3322f10/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/396ec9e0c6096a383b6825a959cf12e798c0e03938d8d1145f3ee83eb3322f10/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/396ec9e0c6096a383b6825a959cf12e798c0e03938d8d1145f3ee83eb3322f10/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/396ec9e0c6096a383b6825a959cf12e798c0e03938d8d1145f3ee83eb3322f10/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : ClickHouse node_0 started (cluster.py:3220, start) 2025-04-04 18:33:24 [ 633 ] DEBUG : get_instance_ip instance_name=node_1 (cluster.py:2082, get_instance_ip) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : get_instance_ip instance_name=node_1 (cluster.py:2092, get_instance_global_ipv6) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : Waiting for ClickHouse start in node_1, ip: 172.16.1.3... (cluster.py:3216, start) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/be22fa31110845ebb812e831f73b741964d47efa7dad3b160a20d62667a87be2/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : ClickHouse node_1 started (cluster.py:3220, start) 2025-04-04 18:33:24 [ 633 ] DEBUG : get_instance_ip instance_name=node_2 (cluster.py:2082, get_instance_ip) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : get_instance_ip instance_name=node_2 (cluster.py:2092, get_instance_global_ipv6) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : Waiting for ClickHouse start in node_2, ip: 172.16.1.5... (cluster.py:3216, start) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageazureblobstoragecluster-node_2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/2f0d1290d597a60407a196df58c1690dce8a07974d67224c055c0e0bdbd8379e/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : ClickHouse node_2 started (cluster.py:3220, start) ----------------------------- Captured stdout call ----------------------------- 1,"a" 2,"b" 1 a 2 b 1 a 2 b 1 a 2 b ------------------------------ Captured log call ------------------------------- 2025-04-04 18:33:24 [ 633 ] DEBUG : Executing query INSERT INTO TABLE FUNCTION azureBlobStorage('http://azurite1:30000/devstoreaccount1', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1','Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV', 'auto', 'key UInt64, data String') VALUES (1, 'a'), (2, 'b') on node_0 (cluster.py:3677, query) 2025-04-04 18:33:24 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30000/devstoreaccount1/cont/test_cluster_select_all.csv' Request method: 'GET' Request headers: 'x-ms-range': 'REDACTED' 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '4b25225a-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:33:24 [ 633 ] DEBUG : Starting new HTTP connection (1): 127.0.0.1:30000 (connectionpool.py:245, _new_conn) 2025-04-04 18:33:24 [ 633 ] DEBUG : http://127.0.0.1:30000 "GET /devstoreaccount1/cont/test_cluster_select_all.csv HTTP/1.1" 206 12 (connectionpool.py:547, _make_request) 2025-04-04 18:33:24 [ 633 ] INFO : Response status: 206 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'last-modified': 'Fri, 04 Apr 2025 18:33:24 GMT' 'x-ms-creation-time': 'REDACTED' 'content-length': '12' 'content-type': 'application/octet-stream' 'content-range': 'REDACTED' 'etag': '"0x1DA5F3FE38BAAB0"' 'x-ms-blob-type': 'REDACTED' 'x-ms-lease-state': 'REDACTED' 'x-ms-lease-status': 'REDACTED' 'x-ms-client-request-id': '4b25225a-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'e1bcd138-554e-47a9-8289-fd02bd30aa31' 'x-ms-version': 'REDACTED' 'accept-ranges': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:33:24 GMT' 'x-ms-server-encrypted': 'REDACTED' 'x-ms-blob-content-md5': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' (_universal.py:550, on_response) 2025-04-04 18:33:24 [ 633 ] DEBUG : Executing query SELECT * from azureBlobStorage('http://azurite1:30000/devstoreaccount1', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1','Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV','auto') on node_0 (cluster.py:3677, query) 2025-04-04 18:33:24 [ 633 ] DEBUG : Executing query SELECT * from azureBlobStorageCluster('simple_cluster', 'http://azurite1:30000/devstoreaccount1', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1','Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV','auto') on node_0 (cluster.py:3677, query) 2025-04-04 18:33:24 [ 633 ] DEBUG : Executing query SELECT * from azureBlobStorage('http://azurite1:30000/devstoreaccount1', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1','Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV','auto') SETTINGS object_storage_cluster='simple_cluster' on node_0 (cluster.py:3677, query) 2025-04-04 18:33:25 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS azure_engine_table_single_node; CREATE TABLE azure_engine_table_single_node (key UInt64, data String) ENGINE=AzureBlobStorage( 'http://azurite1:30000/devstoreaccount1', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV', 'auto' ) on node_0 (cluster.py:3677, query) 2025-04-04 18:33:25 [ 633 ] DEBUG : Executing query SELECT * FROM azure_engine_table_single_node on node_0 (cluster.py:3677, query) 2025-04-04 18:33:25 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS azure_engine_table_distributed; CREATE TABLE azure_engine_table_distributed (key UInt64, data String) ENGINE=AzureBlobStorage( 'http://azurite1:30000/devstoreaccount1', 'cont', 'test_cluster_select_all.csv', 'devstoreaccount1', 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==', 'CSV', 'auto' ) SETTINGS object_storage_cluster='simple_cluster' on node_0 (cluster.py:3677, query) 2025-04-04 18:33:25 [ 633 ] DEBUG : Executing query SELECT * FROM azure_engine_table_distributed on node_0 (cluster.py:3677, query) ---------------------------- Captured log teardown ----------------------------- 2025-04-04 18:33:25 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env --project-name rootteststorageazureblobstoragecluster --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/docker-compose.yml stop --timeout 20] (cluster.py:122, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-04 18:33:48 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/.env --project-name rootteststorageazureblobstoragecluster --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_1/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_azure_blob_storage/_instances-cluster-2/node_2/docker-compose.yml down --volumes] (cluster.py:122, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_0-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_2-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-node_1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Container rootteststorageazureblobstoragecluster-azurite1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Volume rootteststorageazureblobstoragecluster_data1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Network rootteststorageazureblobstoragecluster_default Removing (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Volume rootteststorageazureblobstoragecluster_data1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stderr: Network rootteststorageazureblobstoragecluster_default Removed (cluster.py:148, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Cleanup called (cluster.py:894, cleanup) 2025-04-04 18:33:49 [ 633 ] DEBUG : Docker networks for project rootteststorageazureblobstoragecluster are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-04 18:33:49 [ 633 ] DEBUG : Docker containers for project rootteststorageazureblobstoragecluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-04 18:33:49 [ 633 ] DEBUG : Docker volumes for project rootteststorageazureblobstoragecluster are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-04 18:33:49 [ 633 ] DEBUG : Command:[docker container list --all --filter name='^/rootteststorageazureblobstoragecluster-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup) 2025-04-04 18:33:49 [ 633 ] DEBUG : No running containers for project: rootteststorageazureblobstoragecluster (cluster.py:922, cleanup) 2025-04-04 18:33:49 [ 633 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup) 2025-04-04 18:33:49 [ 633 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup) 2025-04-04 18:33:49 [ 633 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Images pruned (cluster.py:947, cleanup) 2025-04-04 18:33:49 [ 633 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup) 2025-04-04 18:33:49 [ 633 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-04 18:33:49 [ 633 ] DEBUG : Volumes pruned: 1 (cluster.py:958, cleanup) _____________________ test_cluster_table_function[azure-1] _____________________ started_cluster = format_version = '1', storage_type = 'azure' @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs"]) def test_cluster_table_function(started_cluster, format_version, storage_type): if is_arm() and storage_type == "hdfs": pytest.skip("Disabled test IcebergHDFS for aarch64") instance = started_cluster.instances["node1"] spark = started_cluster.spark_session TABLE_NAME = ( "test_iceberg_cluster_" + format_version + "_" + storage_type + "_" + get_uuid_str() ) def add_df(mode): write_iceberg_from_df( spark, generate_data(spark, 0, 100), TABLE_NAME, mode=mode, format_version=format_version, ) files = default_upload_directory( started_cluster, storage_type, f"/iceberg_data/default/{TABLE_NAME}/", f"/iceberg_data/default/{TABLE_NAME}/", ) logging.info(f"Adding another dataframe. result files: {files}") return files files = add_df(mode="overwrite") for i in range(1, len(started_cluster.instances)): files = add_df(mode="append") logging.info(f"Setup complete. files: {files}") assert len(files) == 5 + 4 * (len(started_cluster.instances) - 1) clusters = instance.query(f"SELECT * FROM system.clusters") logging.info(f"Clusters setup: {clusters}") # Regular Query only node1 table_function_expr = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True ) select_regular = ( instance.query(f"SELECT * FROM {table_function_expr}").strip().split() ) # Cluster Query with node1 as coordinator table_function_expr_cluster = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True, run_on_cluster=True, ) query_id_cluster = str(uuid.uuid4()) select_cluster = ( instance.query( f"SELECT * FROM {table_function_expr_cluster}", query_id=query_id_cluster ) .strip() .split() ) # Cluster Query with node1 as coordinator with alternative syntax query_id_cluster_alt_syntax = str(uuid.uuid4()) select_cluster_alt_syntax = ( instance.query( f""" SELECT * FROM {table_function_expr} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_cluster_alt_syntax, ) .strip() .split() ) create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster, object_storage_cluster='cluster_simple') query_id_cluster_table_engine = str(uuid.uuid4()) select_cluster_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_cluster_table_engine, ) .strip() .split() ) select_remote_cluster = ( instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") .strip() .split() ) instance.query(f"DROP TABLE IF EXISTS `{TABLE_NAME}` SYNC") create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster) query_id_pure_table_engine = str(uuid.uuid4()) select_pure_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_pure_table_engine, ) .strip() .split() ) query_id_pure_table_engine_cluster = str(uuid.uuid4()) select_pure_table_engine_cluster = ( instance.query( f""" SELECT * FROM {TABLE_NAME} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_pure_table_engine_cluster, ) .strip() .split() ) # Simple size check assert len(select_regular) == 600 assert len(select_cluster) == 600 assert len(select_cluster_alt_syntax) == 600 > assert len(select_cluster_table_engine) == 600 E AssertionError: assert 1800 == 600 E + where 1800 = len(['0', '1', '1', '2', '2', '3', ...]) test_storage_iceberg/test.py:747: AssertionError ---------------------------- Captured stdout setup ----------------------------- 25/04/04 18:33:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- Picked up JAVA_TOOL_OPTIONS: -Djdk.attach.allowAttachSelf=true Picked up JAVA_TOOL_OPTIONS: -Djdk.attach.allowAttachSelf=true Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). ------------------------------ Captured log setup ------------------------------ 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV HOSTNAME 52698a6efbf1 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV SHLVL 0 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV HOME /root (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV OLDPWD / (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_HELPER_TAG 5dc43a6382f0 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV PYTEST_ADDOPTS -rfEps --run-id=2 --color=no --durations=0 test_storage_azure_blob_storage/test_cluster.py::test_select_all 'test_storage_iceberg/test.py::test_cluster_table_function[azure-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[azure-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-2]' -vvv (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV TZ Etc/UTC (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV DOCKER_BASE_TAG 6712d5cc610d (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 2 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : ENV PYTEST_CURRENT_TEST test_storage_iceberg/test.py::test_cluster_table_function[azure-1] (setup) (cluster.py:450, __init__) 2025-04-04 18:33:49 [ 633 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:774, __init__) 2025-04-04 18:33:51 [ 633 ] DEBUG : GatewayClient.address is deprecated and will be removed in version 1.0. Use GatewayParameters instead. (java_gateway.py:163, deprecated) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: A 872d8af20d647d641be91feca50cdba2b644252e9b01dd434129e44d8a7d67a5 (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.SparkConf e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.api.java.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.api.python.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.ml.python.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.mllib.api.python.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.resource.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.sql.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.sql.api.python.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj org.apache.spark.sql.hive.* e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: j i rj scala.Tuple2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: r u SparkConf rj e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.SparkConf (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: i org.apache.spark.SparkConf bTrue e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro0 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 set sspark.app.name sspark_test e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro1 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 set sspark.master slocal e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro2 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 contains sspark.serializer.objectStreamReset e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 set sspark.serializer.objectStreamReset s100 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro3 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 contains sspark.rdd.compress e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 set sspark.rdd.compress sTrue e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro4 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 contains sspark.master e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 contains sspark.app.name e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 contains sspark.master e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 get sspark.master e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 contains sspark.app.name e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 get sspark.app.name e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 contains sspark.home e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o0 getAll e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yto5 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i0 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro6 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o6 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.master (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o6 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro7 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o7 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.app.name (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o7 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o8 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.rdd.compress (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o8 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysTrue (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i3 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro9 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o9 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.serializer.objectStreamReset (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o9 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ys100 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i4 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro10 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o10 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.submit.pyFiles (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o10 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro11 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o11 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.app.submitTime (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o11 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ys1743791631082 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i6 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro12 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o12 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.submit.deployMode (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o12 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysclient (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a g o5 i7 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yro13 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o13 _1 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ysspark.ui.showConsoleProgress (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: c o13 _2 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ystrue (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: r u JavaSparkContext rj e (clientserver.py:501, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.java.JavaSparkContext (clientserver.py:512, send_command) 2025-04-04 18:33:51 [ 633 ] DEBUG : Command to send: i org.apache.spark.api.java.JavaSparkContext ro0 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: A 872d8af20d647d641be91feca50cdba2b644252e9b01dd434129e44d8a7d67a5 (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o1 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o2 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o3 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o4 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o5 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o6 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o7 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o8 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o9 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o10 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o11 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o12 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: m d o13 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro14 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro15 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o15 conf e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro16 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u PythonAccumulatorV2 rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonAccumulatorV2 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: i org.apache.spark.api.python.PythonAccumulatorV2 s127.0.0.1 i64345 s872d8af20d647d641be91feca50cdba2b644252e9b01dd434129e44d8a7d67a5 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro17 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro18 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o18 register ro17 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils isEncryptionEnabled e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils isEncryptionEnabled ro14 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout ro14 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yL15 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getSparkBufferSize e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getSparkBufferSize ro14 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yi65536 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark.SparkFiles rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.SparkFiles (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ys/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0/userFiles-7556ac45-1b66-482d-a34a-c7af0d068289 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o16 get sspark.submit.pyFiles s e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.util.Utils getLocalDir e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro19 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o19 conf e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro20 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils getLocalDir ro20 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ys/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.util.Utils createTempDir e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils createTempDir s/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0 spyspark e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro21 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o21 getAbsolutePath e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ys/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0/pyspark-f09cb669-6190-4e21-9122-8d99d8a21412 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o16 get sspark.python.profile sfalse e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ysfalse (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro22 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o22 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro23 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yao24 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o24 put sspark.app.name sspark_test e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o24 put sspark.master slocal e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: i org.apache.spark.sql.SparkSession ro23 ro24 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yro25 (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setDefaultSession ro25 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setActiveSession ro25 e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c o14 stop e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession clearDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession clearDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession clearActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession clearActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:52 [ 633 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-04 18:33:52 [ 633 ] DEBUG : HDFS BASE CMD:{self.base_hdfs_cmd)} (cluster.py:1269, setup_hdfs_cmd) 2025-04-04 18:33:52 [ 633 ] DEBUG : Cluster name: project_name:rootteststorageiceberg. Added instance name:node1 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env', '--project-name', 'rootteststorageiceberg', '--file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-04 18:33:52 [ 633 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-04 18:33:52 [ 633 ] DEBUG : Cluster name: project_name:rootteststorageiceberg. Added instance name:node2 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env', '--project-name', 'rootteststorageiceberg', '--file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml', '--file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-04 18:33:52 [ 633 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-04 18:33:52 [ 633 ] DEBUG : Cluster name: project_name:rootteststorageiceberg. Added instance name:node3 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env', '--project-name', 'rootteststorageiceberg', '--file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml', '--file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-04 18:33:52 [ 633 ] INFO : Starting cluster... (test.py:110, started_cluster) 2025-04-04 18:33:52 [ 633 ] INFO : Running tests in /ClickHouse/tests/integration/test_storage_iceberg/test.py (cluster.py:2793, start) 2025-04-04 18:33:52 [ 633 ] DEBUG : Cluster start called. is_up=False (cluster.py:2800, start) 2025-04-04 18:33:52 [ 633 ] DEBUG : Docker networks for project rootteststorageiceberg are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-04 18:33:52 [ 633 ] DEBUG : Docker containers for project rootteststorageiceberg are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-04 18:33:52 [ 633 ] DEBUG : Docker volumes for project rootteststorageiceberg are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-04 18:33:52 [ 633 ] DEBUG : Cleanup called (cluster.py:894, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : Docker networks for project rootteststorageiceberg are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-04 18:33:52 [ 633 ] DEBUG : Docker containers for project rootteststorageiceberg are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-04 18:33:52 [ 633 ] DEBUG : Docker volumes for project rootteststorageiceberg are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command:[docker container list --all --filter name='^/rootteststorageiceberg-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check) 2025-04-04 18:33:52 [ 633 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : No running containers for project: rootteststorageiceberg (cluster.py:922, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check) 2025-04-04 18:33:52 [ 633 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-04 18:33:52 [ 633 ] DEBUG : Images pruned (cluster.py:947, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check) 2025-04-04 18:33:52 [ 633 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-04 18:33:52 [ 633 ] DEBUG : Volumes pruned: 1 (cluster.py:958, cleanup) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup directory for instance: node1 (cluster.py:2813, start) 2025-04-04 18:33:52 [ 633 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/query_log.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/cluster.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/named_collections.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/filesystem_caches.xml'] to /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/configs/config.d (cluster.py:4752, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/database (cluster.py:4769, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/logs (cluster.py:4780, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4864, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup directory for instance: node2 (cluster.py:2813, start) 2025-04-04 18:33:52 [ 633 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/query_log.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/cluster.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/named_collections.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/filesystem_caches.xml'] to /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/configs/config.d (cluster.py:4752, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/database (cluster.py:4769, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/logs (cluster.py:4780, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4864, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup directory for instance: node3 (cluster.py:2813, start) 2025-04-04 18:33:52 [ 633 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/query_log.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/cluster.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/named_collections.xml', '/ClickHouse/tests/integration/test_storage_iceberg/configs/config.d/filesystem_caches.xml'] to /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/configs/config.d (cluster.py:4752, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/database (cluster.py:4769, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/logs (cluster.py:4780, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4864, create_dir) 2025-04-04 18:33:52 [ 633 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'HDFS_HOST': 'hdfs1', 'HDFS_NAME_PORT': '50070', 'HDFS_DATA_PORT': '50075', 'HDFS_LOGS': '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/hdfs/logs', 'HDFS_FS': 'bind', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_storage_iceberg/_instances-2/resolver', 'RESOLVER_LOGS_FS': 'bind', 'AZURITE_PORT': '30001', 'AZURITE_STORAGE_ACCOUNT_URL': 'http://azurite1:30001/devstoreaccount1', 'AZURITE_CONNECTION_STRING': 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite1:30001/devstoreaccount1;'} stored in /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env (cluster.py:97, _create_env_file) 2025-04-04 18:33:52 [ 633 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-04 18:33:52 [ 633 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-04 18:33:52 [ 633 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-04 18:33:52 [ 633 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-04 18:33:52 [ 633 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2025-04-04 18:33:52 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --project-name rootteststorageiceberg --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/docker-compose.yml pull] (cluster.py:122, run_and_check) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o24 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o15 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o18 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o19 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o20 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o21 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o22 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Command to send: m d o23 e (clientserver.py:501, send_command) 2025-04-04 18:33:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: proxy2 Skipped - Image is already being pulled by proxy1 (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: node2 Skipped - Image is already being pulled by node1 (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: node3 Skipped - Image is already being pulled by node1 (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: hdfs1 Pulling (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: azurite1 Pulling (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: proxy1 Pulling (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: minio1 Pulling (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: resolver Pulling (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: node1 Pulling (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: node1 Pulled (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: resolver Pulled (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: proxy1 Pulled (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: hdfs1 Pulled (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: minio1 Pulled (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: azurite1 Pulled (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Setup HDFS (cluster.py:3064, start) 2025-04-04 18:34:54 [ 633 ] DEBUG : Command:[docker compose --project-name rootteststorageiceberg --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml --verbose up -d] (cluster.py:122, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:34:54Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: Network rootteststorageiceberg_default Creating (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: Network rootteststorageiceberg_default Created (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:34:54Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:34:54Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:34:54 [ 633 ] DEBUG : get_instance_ip instance_name=hdfs1 (cluster.py:2082, get_instance_ip) 2025-04-04 18:34:54 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-hdfs1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:34:54 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:34:54 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:34:54 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:34:54 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:34:55 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:34:55 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:34:55 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:34:55 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:34:56 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:34:56 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:34:56 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:34:56 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:34:57 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:34:57 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:34:57 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:34:57 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:34:58 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:34:58 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:34:58 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:34:58 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:34:59 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:34:59 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:34:59 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:34:59 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:35:00 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:00 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:00 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:00 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:35:01 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:01 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:01 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:01 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:35:02 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:02 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:02 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:02 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:35:03 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:03 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:03 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:03 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 497, in _make_request conn.request( File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 395, in request self.endheaders() File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 976, in send self.connect() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 243, in connect self.sock = self._new_conn() File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 218, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 132, in req_wrapper response_data = func(**kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 130, in put return request("put", url, data=data, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.16.1.2', port=50070): Max retries exceeded with url: /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) 2025-04-04 18:35:04 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:06 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RetriableException","javaClassName":"org.apache.hadoop.ipc.RetriableException","message":"Namenode is in startup mode"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:04 GMT, Fri, 04 Apr 2025 18:35:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:04 GMT, Fri, 04 Apr 2025 18:35:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:06 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:07 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:07 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:07 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:07 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RetriableException","javaClassName":"org.apache.hadoop.ipc.RetriableException","message":"Namenode is in startup mode"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:07 GMT, Fri, 04 Apr 2025 18:35:07 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:07 GMT, Fri, 04 Apr 2025 18:35:07 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:07 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:08 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true 2025-04-04 18:35:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:09 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RetriableException","javaClassName":"org.apache.hadoop.ipc.RetriableException","message":"Namenode is in startup mode"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:09 GMT, Fri, 04 Apr 2025 18:35:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:09 GMT, Fri, 04 Apr 2025 18:35:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:09 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:10 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RetriableException","javaClassName":"org.apache.hadoop.ipc.RetriableException","message":"Namenode is in startup mode"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:10 GMT, Fri, 04 Apr 2025 18:35:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:10 GMT, Fri, 04 Apr 2025 18:35:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:10 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:11 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true 2025-04-04 18:35:12 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:12 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:12 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:12 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:12 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RetriableException","javaClassName":"org.apache.hadoop.ipc.RetriableException","message":"Namenode is in startup mode"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:12 GMT, Fri, 04 Apr 2025 18:35:12 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:12 GMT, Fri, 04 Apr 2025 18:35:12 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:12 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:13 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:13 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:13 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:13 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RetriableException","javaClassName":"org.apache.hadoop.ipc.RetriableException","message":"Namenode is in startup mode"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:13 GMT, Fri, 04 Apr 2025 18:35:13 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:13 GMT, Fri, 04 Apr 2025 18:35:13 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:13 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:14 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true 2025-04-04 18:35:15 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:15 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:15 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:15 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:15 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"IOException","javaClassName":"java.io.IOException","message":"Failed to find datanode, suggest to check cluster health."}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:15 GMT, Fri, 04 Apr 2025 18:35:15 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:15 GMT, Fri, 04 Apr 2025 18:35:15 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:15 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:16 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:16 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:16 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:16 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"IOException","javaClassName":"java.io.IOException","message":"Failed to find datanode, suggest to check cluster health."}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:16 GMT, Fri, 04 Apr 2025 18:35:16 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:16 GMT, Fri, 04 Apr 2025 18:35:16 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:16 [ 633 ] ERROR : unexpected response_data.status_code 403 != 307 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:17 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 211, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true 2025-04-04 18:35:18 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:18 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:18 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:18 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:18 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:18 GMT, Fri, 04 Apr 2025 18:35:18 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:18 GMT, Fri, 04 Apr 2025 18:35:18 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:18 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:18 GMT, Fri, 04 Apr 2025 18:35:18 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:18 GMT, Fri, 04 Apr 2025 18:35:18 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:18 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:18 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:19 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:19 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 28 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:18 GMT, Fri, 04 Apr 2025 18:35:18 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:18 GMT, Fri, 04 Apr 2025 18:35:18 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:19 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:20 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:20 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:20 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:20 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 27 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:20 GMT, Fri, 04 Apr 2025 18:35:20 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:20 GMT, Fri, 04 Apr 2025 18:35:20 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:20 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:21 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:22 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:22 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:22 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:22 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:22 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:22 GMT, Fri, 04 Apr 2025 18:35:22 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:22 GMT, Fri, 04 Apr 2025 18:35:22 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:22 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:22 GMT, Fri, 04 Apr 2025 18:35:22 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:22 GMT, Fri, 04 Apr 2025 18:35:22 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:22 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:22 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:22 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:22 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 25 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:22 GMT, Fri, 04 Apr 2025 18:35:22 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:22 GMT, Fri, 04 Apr 2025 18:35:22 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:22 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:23 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:23 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:23 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:23 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 24 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:23 GMT, Fri, 04 Apr 2025 18:35:23 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:23 GMT, Fri, 04 Apr 2025 18:35:23 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:23 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:24 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:25 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:25 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:25 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:25 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:25 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:25 GMT, Fri, 04 Apr 2025 18:35:25 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:25 GMT, Fri, 04 Apr 2025 18:35:25 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:25 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:25 GMT, Fri, 04 Apr 2025 18:35:25 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:25 GMT, Fri, 04 Apr 2025 18:35:25 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:25 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:25 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:25 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:25 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 22 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:25 GMT, Fri, 04 Apr 2025 18:35:25 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:25 GMT, Fri, 04 Apr 2025 18:35:25 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:25 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:26 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:26 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:26 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:26 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 21 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:26 GMT, Fri, 04 Apr 2025 18:35:26 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:26 GMT, Fri, 04 Apr 2025 18:35:26 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:26 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:27 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:28 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:28 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:28 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:28 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:28 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:28 GMT, Fri, 04 Apr 2025 18:35:28 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:28 GMT, Fri, 04 Apr 2025 18:35:28 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:28 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:28 GMT, Fri, 04 Apr 2025 18:35:28 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:28 GMT, Fri, 04 Apr 2025 18:35:28 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:28 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:28 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:28 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:28 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 19 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:28 GMT, Fri, 04 Apr 2025 18:35:28 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:28 GMT, Fri, 04 Apr 2025 18:35:28 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:28 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:29 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:29 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:29 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:29 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 18 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:29 GMT, Fri, 04 Apr 2025 18:35:29 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:29 GMT, Fri, 04 Apr 2025 18:35:29 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:29 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:30 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:31 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:31 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:31 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:31 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:31 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:31 GMT, Fri, 04 Apr 2025 18:35:31 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:31 GMT, Fri, 04 Apr 2025 18:35:31 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:31 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:31 GMT, Fri, 04 Apr 2025 18:35:31 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:31 GMT, Fri, 04 Apr 2025 18:35:31 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:31 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:31 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:31 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:31 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 16 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:31 GMT, Fri, 04 Apr 2025 18:35:31 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:31 GMT, Fri, 04 Apr 2025 18:35:31 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:31 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:32 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:32 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:32 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:32 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 14 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:32 GMT, Fri, 04 Apr 2025 18:35:32 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:32 GMT, Fri, 04 Apr 2025 18:35:32 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:32 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:33 [ 633 ] DEBUG : Command to send: m d o0 e (clientserver.py:501, send_command) 2025-04-04 18:35:33 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:33 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:34 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:34 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:34 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:34 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:34 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:34 GMT, Fri, 04 Apr 2025 18:35:34 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:34 GMT, Fri, 04 Apr 2025 18:35:34 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:34 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:34 GMT, Fri, 04 Apr 2025 18:35:34 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:34 GMT, Fri, 04 Apr 2025 18:35:34 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:34 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:34 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:34 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:34 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 12 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:34 GMT, Fri, 04 Apr 2025 18:35:34 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:34 GMT, Fri, 04 Apr 2025 18:35:34 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:34 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:35 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:35 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:35 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:35 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 11 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:35 GMT, Fri, 04 Apr 2025 18:35:35 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:35 GMT, Fri, 04 Apr 2025 18:35:35 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:35 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:36 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:37 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:37 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:37 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:37 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:37 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:37 GMT, Fri, 04 Apr 2025 18:35:37 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:37 GMT, Fri, 04 Apr 2025 18:35:37 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:37 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:37 GMT, Fri, 04 Apr 2025 18:35:37 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:37 GMT, Fri, 04 Apr 2025 18:35:37 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:37 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:37 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:37 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:37 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 9 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:37 GMT, Fri, 04 Apr 2025 18:35:37 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:37 GMT, Fri, 04 Apr 2025 18:35:37 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:37 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:38 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:38 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:38 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:38 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 8 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:38 GMT, Fri, 04 Apr 2025 18:35:38 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:38 GMT, Fri, 04 Apr 2025 18:35:38 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:38 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:39 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:40 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:40 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:40 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:40 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:40 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:40 GMT, Fri, 04 Apr 2025 18:35:40 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:40 GMT, Fri, 04 Apr 2025 18:35:40 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:40 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:40 GMT, Fri, 04 Apr 2025 18:35:40 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:40 GMT, Fri, 04 Apr 2025 18:35:40 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:40 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:40 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:40 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:40 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 6 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:40 GMT, Fri, 04 Apr 2025 18:35:40 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:40 GMT, Fri, 04 Apr 2025 18:35:40 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:40 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:41 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:41 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:41 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:41 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 5 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:41 GMT, Fri, 04 Apr 2025 18:35:41 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:41 GMT, Fri, 04 Apr 2025 18:35:41 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:41 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:42 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:43 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:43 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:43 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:43 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:43 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:43 GMT, Fri, 04 Apr 2025 18:35:43 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:43 GMT, Fri, 04 Apr 2025 18:35:43 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:43 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:43 GMT, Fri, 04 Apr 2025 18:35:43 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:43 GMT, Fri, 04 Apr 2025 18:35:43 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:43 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:43 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:43 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:43 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 3 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:43 GMT, Fri, 04 Apr 2025 18:35:43 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:43 GMT, Fri, 04 Apr 2025 18:35:43 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:43 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:44 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:44 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:44 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:44 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 2 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:44 GMT, Fri, 04 Apr 2025 18:35:44 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:44 GMT, Fri, 04 Apr 2025 18:35:44 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:44 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:45 [ 633 ] ERROR : Can't connect to HDFS or preparations are not done yet 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root (cluster.py:2572, wait_hdfs_to_start) Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2565, in wait_hdfs_to_start self.hdfs_api.write_data("/somefilewithrandomname222", "1") File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 242, in write_data response = self.req_wrapper( File "/ClickHouse/tests/integration/helpers/hdfs_api.py", line 143, in req_wrapper response_data.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root 2025-04-04 18:35:46 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /somefilewithrandomname222 user:root, principal:None (hdfs_api.py:194, write_data) 2025-04-04 18:35:46 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/somefilewithrandomname222?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:46 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:46 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:46 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:46 GMT, Fri, 04 Apr 2025 18:35:46 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:46 GMT, Fri, 04 Apr 2025 18:35:46 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:46 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:46 GMT, Fri, 04 Apr 2025 18:35:46 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:46 GMT, Fri, 04 Apr 2025 18:35:46 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:228, write_data) 2025-04-04 18:35:46 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:46 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:46 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 403 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:46 [ 633 ] DEBUG : response_data:b'{"RemoteException":{"exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException","message":"Cannot create file/somefilewithrandomname222. Name node is in safe mode.\\nThe reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The number of live datanodes 1 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 0 seconds.\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2630)\\n\\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2519)\\n\\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:566)\\n\\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)\\n\\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\\n\\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)\\n\\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)\\n\\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)\\n\\tat java.security.AccessController.doPrivileged(Native Method)\\n\\tat javax.security.auth.Subject.doAs(Subject.java:415)\\n\\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\\n\\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)\\n"}}' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:46 GMT, Fri, 04 Apr 2025 18:35:46 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:46 GMT, Fri, 04 Apr 2025 18:35:46 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:46 [ 633 ] ERROR : unexpected response_data.status_code 403 != 201 (hdfs_api.py:139, req_wrapper) 2025-04-04 18:35:47 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/somefilewithrandomname222', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:35:47 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:48 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/somefilewithrandomname222?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Fsomefilewithrandomname222&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:48 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:47 GMT, Fri, 04 Apr 2025 18:35:47 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:47 GMT, Fri, 04 Apr 2025 18:35:47 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/somefilewithrandomname222', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:35:48 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:35:47 GMT, Fri, 04 Apr 2025 18:35:47 GMT', 'Date': 'Fri, 04 Apr 2025 18:35:47 GMT, Fri, 04 Apr 2025 18:35:47 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/somefilewithrandomname222', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:253, write_data) 2025-04-04 18:35:48 [ 633 ] DEBUG : Connected to HDFS and SafeMode disabled! (cluster.py:2566, wait_hdfs_to_start) 2025-04-04 18:35:48 [ 633 ] INFO : Trying to create Minio instance by command docker compose --project-name rootteststorageiceberg --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d (cluster.py:3132, start) 2025-04-04 18:35:48 [ 633 ] DEBUG : Command:[docker compose --project-name rootteststorageiceberg --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] (cluster.py:122, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:48Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Volume "rootteststorageiceberg_data1-1" Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Volume "rootteststorageiceberg_data1-1" Created (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:48Z" level=warning msg="Found orphan containers ([rootteststorageiceberg-hdfs1-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: proxy1 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: proxy2 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:49Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:49Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:35:49 [ 633 ] INFO : Trying to connect to Minio... (cluster.py:3138, start) 2025-04-04 18:35:49 [ 633 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:2082, get_instance_ip) 2025-04-04 18:35:49 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-minio1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:49 [ 633 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:2082, get_instance_ip) 2025-04-04 18:35:49 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-proxy1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:49 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.5:9001 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:49 [ 633 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-04-04 18:35:49 [ 633 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-04-04 18:35:49 [ 633 ] DEBUG : Starting new HTTP connection (2): 172.16.1.5:9001 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:49 [ 633 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-04-04 18:35:49 [ 633 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-04-04 18:35:49 [ 633 ] DEBUG : Starting new HTTP connection (3): 172.16.1.5:9001 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:49 [ 633 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-04-04 18:35:49 [ 633 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-04-04 18:35:49 [ 633 ] DEBUG : Starting new HTTP connection (4): 172.16.1.5:9001 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:49 [ 633 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.1.5', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2637, wait_minio_to_start) 2025-04-04 18:35:50 [ 633 ] DEBUG : Starting new HTTP connection (5): 172.16.1.5:9001 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:50 [ 633 ] DEBUG : http://172.16.1.5:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:50 [ 633 ] DEBUG : Connected to Minio. (cluster.py:2617, wait_minio_to_start) 2025-04-04 18:35:50 [ 633 ] DEBUG : http://172.16.1.5:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:50 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:50 [ 633 ] DEBUG : S3 bucket 'root' created (cluster.py:2632, wait_minio_to_start) 2025-04-04 18:35:50 [ 633 ] DEBUG : http://172.16.1.5:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:50 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:50 [ 633 ] DEBUG : S3 bucket 'root2' created (cluster.py:2632, wait_minio_to_start) 2025-04-04 18:35:50 [ 633 ] INFO : Trying to create Azurite instance by command docker compose --project-name rootteststorageiceberg --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --verbose up -d (cluster.py:3143, start) 2025-04-04 18:35:50 [ 633 ] DEBUG : Command:[docker compose --project-name rootteststorageiceberg --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --verbose up -d] (cluster.py:122, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:50Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:50Z" level=warning msg="Found orphan containers ([rootteststorageiceberg-resolver-1 rootteststorageiceberg-minio1-1 rootteststorageiceberg-proxy2-1 rootteststorageiceberg-proxy1-1 rootteststorageiceberg-hdfs1-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:50Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] DEBUG : Stderr:time="2025-04-04T18:35:50Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-04 18:35:50 [ 633 ] INFO : Trying to connect to Azurite (cluster.py:3157, start) 2025-04-04 18:35:51 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/?restype=REDACTED&comp=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a2c4d26c-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:35:51 [ 633 ] DEBUG : Starting new HTTP connection (1): 127.0.0.1:30001 (connectionpool.py:245, _new_conn) 2025-04-04 18:35:51 [ 633 ] DEBUG : http://127.0.0.1:30001 "GET /devstoreaccount1/?restype=account&comp=properties HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:51 [ 633 ] INFO : Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-client-request-id': 'a2c4d26c-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'd7935af8-391c-4908-ad8f-7c4184f8d987' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:51 GMT' 'x-ms-sku-name': 'REDACTED' 'x-ms-account-kind': 'REDACTED' 'x-ms-is-hns-enabled': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:51 [ 633 ] DEBUG : {'client_request_id': 'a2c4d26c-1183-11f0-b12d-0242ac110002', 'request_id': 'd7935af8-391c-4908-ad8f-7c4184f8d987', 'version': '2025-05-05', 'date': datetime.datetime(2025, 4, 4, 18, 35, 51, tzinfo=datetime.timezone.utc), 'sku_name': 'Standard_RAGRS', 'account_kind': 'StorageV2', 'is_hns_enabled': False} (cluster.py:2665, wait_azurite_to_start) 2025-04-04 18:35:51 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/?comp=REDACTED&prefix=REDACTED&include=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a2cbab50-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:35:51 [ 633 ] DEBUG : http://127.0.0.1:30001 "GET /devstoreaccount1/?comp=list&prefix=azurite-container&include= HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:51 [ 633 ] INFO : Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-client-request-id': 'a2cbab50-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '7ff2f2df-a9d5-412d-8f41-2017262c8b17' 'x-ms-version': 'REDACTED' 'content-type': 'application/xml' 'Date': 'Fri, 04 Apr 2025 18:35:51 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' (_universal.py:550, on_response) 2025-04-04 18:35:51 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/azurite-container?restype=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a2cd20ac-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:35:51 [ 633 ] DEBUG : http://127.0.0.1:30001 "GET /devstoreaccount1/azurite-container?restype=container HTTP/1.1" 404 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:51 [ 633 ] INFO : Response status: 404 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-error-code': 'ContainerNotFound' 'x-ms-request-id': '951e7778-ac52-4f7d-87ca-ec62782b6534' 'content-type': 'application/xml' 'Date': 'Fri, 04 Apr 2025 18:35:51 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' (_universal.py:550, on_response) 2025-04-04 18:35:51 [ 633 ] DEBUG : azurite container 'azurite-container' doesn't exist, creating it (cluster.py:2687, wait_azurite_to_start) 2025-04-04 18:35:51 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/azurite-container?restype=REDACTED' Request method: 'PUT' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a2ce6200-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:35:51 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/azurite-container?restype=container HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:51 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1D295B242638440"' 'last-modified': 'Fri, 04 Apr 2025 18:35:51 GMT' 'x-ms-client-request-id': 'a2ce6200-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'cc406bc7-8772-4efb-9d15-fa2dd8569d66' 'x-ms-version': 'REDACTED' 'Date': 'Fri, 04 Apr 2025 18:35:51 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:51 [ 633 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --project-name rootteststorageiceberg --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/docker-compose.yml up -d --no-recreate') (cluster.py:3200, start) 2025-04-04 18:35:51 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --project-name rootteststorageiceberg --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/docker-compose.yml up -d --no-recreate] (cluster.py:122, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Running (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Running (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Running (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Running (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Running (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Running (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Creating (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Created (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Starting (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Started (cluster.py:148, run_and_check) 2025-04-04 18:35:52 [ 633 ] DEBUG : ClickHouse instance created (cluster.py:3208, start) 2025-04-04 18:35:52 [ 633 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2082, get_instance_ip) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2092, get_instance_global_ipv6) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.1.9... (cluster.py:3216, start) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/3e68cf3ceb84caa34b2955ca4a961cb8e9591f33c642d4d01538e17c575b29d1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/3e68cf3ceb84caa34b2955ca4a961cb8e9591f33c642d4d01538e17c575b29d1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/3e68cf3ceb84caa34b2955ca4a961cb8e9591f33c642d4d01538e17c575b29d1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/3e68cf3ceb84caa34b2955ca4a961cb8e9591f33c642d4d01538e17c575b29d1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/3e68cf3ceb84caa34b2955ca4a961cb8e9591f33c642d4d01538e17c575b29d1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : ClickHouse node1 started (cluster.py:3220, start) 2025-04-04 18:35:52 [ 633 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2082, get_instance_ip) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2092, get_instance_global_ipv6) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.1.8... (cluster.py:3216, start) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/d639d60329832c1353a5d2e8fc380bf658fa8728547301452c57aaba324215c2/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : ClickHouse node2 started (cluster.py:3220, start) 2025-04-04 18:35:52 [ 633 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2082, get_instance_ip) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2092, get_instance_global_ipv6) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.1.10... (cluster.py:3216, start) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/rootteststorageiceberg-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://localhost:None "GET /v1.46/containers/29ba83ceefef78c548599ecf03aae6f31a80004884e5a20c6b03faaf8020d7f1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : ClickHouse node3 started (cluster.py:3220, start) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root?policy= HTTP/1.1" 204 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://172.16.1.5:9001 "GET /root-with-auth?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root-with-auth HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:52 [ 633 ] INFO : S3 bucket created (test.py:114, started_cluster) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: r u SparkConf rj e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.SparkConf (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: i org.apache.spark.SparkConf bTrue e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro26 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.app.name sspark_test e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro27 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.master slocal e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro28 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.sql.catalog.spark_catalog sorg.apache.iceberg.spark.SparkSessionCatalog e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro29 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.sql.catalog.local sorg.apache.iceberg.spark.SparkCatalog e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro30 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.sql.catalog.spark_catalog.type shadoop e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro31 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.sql.catalog.spark_catalog.warehouse s/iceberg_data e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro32 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.sql.extensions sorg.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro33 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 get sspark.executor.allowSparkContext sfalse e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysfalse (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 contains sspark.serializer.objectStreamReset e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.serializer.objectStreamReset s100 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro34 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 contains sspark.rdd.compress e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 set sspark.rdd.compress sTrue e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro35 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 contains sspark.master e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 contains sspark.app.name e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 contains sspark.master e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 get sspark.master e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 contains sspark.app.name e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 get sspark.app.name e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 contains sspark.home e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o26 getAll e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yto36 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i0 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro37 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o37 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.master (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o37 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro38 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o38 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.sql.catalog.local (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o38 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysorg.apache.iceberg.spark.SparkCatalog (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro39 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o39 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.app.name (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o39 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i3 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro40 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o40 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.rdd.compress (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o40 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysTrue (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i4 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro41 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o41 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.serializer.objectStreamReset (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o41 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ys100 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i5 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro42 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o42 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.sql.catalog.spark_catalog.warehouse (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o42 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ys/iceberg_data (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i6 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro43 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o43 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.submit.pyFiles (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o43 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i7 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro44 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o44 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.sql.catalog.spark_catalog (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o44 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysorg.apache.iceberg.spark.SparkSessionCatalog (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i8 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro45 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o45 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.submit.deployMode (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o45 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysclient (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i9 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro46 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o46 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.app.submitTime (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o46 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ys1743791631082 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i10 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro47 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o47 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.sql.catalog.spark_catalog.type (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o47 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yshadoop (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i11 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro48 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o48 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.ui.showConsoleProgress (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o48 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ystrue (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a g o36 i12 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yro49 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o49 _1 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysspark.sql.extensions (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: c o49 _2 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ysorg.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: a e o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !yi13 (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: r u JavaSparkContext rj e (clientserver.py:501, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.java.JavaSparkContext (clientserver.py:512, send_command) 2025-04-04 18:35:52 [ 633 ] DEBUG : Command to send: i org.apache.spark.api.java.JavaSparkContext ro26 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro50 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro51 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o51 conf e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro52 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u PythonAccumulatorV2 rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonAccumulatorV2 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: i org.apache.spark.api.python.PythonAccumulatorV2 s127.0.0.1 i65455 s872d8af20d647d641be91feca50cdba2b644252e9b01dd434129e44d8a7d67a5 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro53 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro54 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o54 register ro53 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils isEncryptionEnabled e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils isEncryptionEnabled ro50 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout ro50 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yL15 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getSparkBufferSize e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getSparkBufferSize ro50 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yi65536 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark.SparkFiles rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.SparkFiles (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ys/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0/userFiles-2c68ba92-02c3-4187-a4fa-21bd2316a17e (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o52 get sspark.submit.pyFiles s e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.util.Utils getLocalDir e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro55 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o55 conf e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro56 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils getLocalDir ro56 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ys/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.util.Utils createTempDir e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils createTempDir s/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0 spyspark e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro57 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o57 getAbsolutePath e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ys/tmp/spark-17e233a6-c99c-4bbb-863a-f0cc3f2a88d0/pyspark-11920067-f574-4e01-a48c-c63a7797de45 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o52 get sspark.python.profile sfalse e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ysfalse (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro58 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o58 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro59 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yao60 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o60 put sspark.app.name sspark_test e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o60 put sspark.master slocal e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o60 put sspark.sql.catalog.spark_catalog sorg.apache.iceberg.spark.SparkSessionCatalog e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o60 put sspark.sql.catalog.local sorg.apache.iceberg.spark.SparkCatalog e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o60 put sspark.sql.catalog.spark_catalog.type shadoop e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o60 put sspark.sql.catalog.spark_catalog.warehouse s/iceberg_data e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o60 put sspark.sql.extensions sorg.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: i org.apache.spark.sql.SparkSession ro59 ro60 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro61 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setDefaultSession e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setDefaultSession ro61 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setActiveSession ro61 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer?restype=REDACTED' Request method: 'PUT' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a391c330-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request (_universal.py:514, on_request) 2025-04-04 18:35:53 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer?restype=container HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:53 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x26A0B79AFBEE780"' 'last-modified': 'Fri, 04 Apr 2025 18:35:53 GMT' 'x-ms-client-request-id': 'a391c330-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '3ee0e02d-9005-4cae-a246-54e8a45bd55f' 'x-ms-version': 'REDACTED' 'Date': 'Fri, 04 Apr 2025 18:35:53 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) ----------------------------- Captured stdout call ----------------------------- 25/04/04 18:35:55 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:55 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:55 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:55 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:57 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:57 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:35:58 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:58 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:58 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:58 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:58 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:58 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:35:59 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:59 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:59 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:59 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:59 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:35:59 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} ----------------------------- Captured stderr call ----------------------------- [Stage 1:> (0 + 1) / 1] ------------------------------ Captured log call ------------------------------- 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yro62 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o62 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o27 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o28 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o29 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o30 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o31 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o32 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o33 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o34 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o35 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o37 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o38 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o39 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o40 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o41 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o42 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o43 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o44 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o45 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o46 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o36 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Command to send: m d o60 e (clientserver.py:501, send_command) 2025-04-04 18:35:53 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Answer received: !yro63 (clientserver.py:512, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Answer received: !ylo64 (clientserver.py:512, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Command to send: c o64 add sa e (clientserver.py:501, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro64 e (clientserver.py:501, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Answer received: !yro65 (clientserver.py:512, send_command) 2025-04-04 18:35:54 [ 633 ] DEBUG : Command to send: c o63 toDF ro65 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro66 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro67 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o67 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro68 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ylo69 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o69 add sb e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro69 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro70 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o68 toDF ro70 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro71 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o71 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro72 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro73 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o73 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro74 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o74 get e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro75 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro76 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yao77 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o76 applyModifiableSettings ro75 ro77 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro78 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o72 cast ro78 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro79 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o71 withColumn sb ro79 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro80 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro81 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro82 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ylo83 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o83 add ro82 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro83 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro84 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro84 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro85 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o81 over ro85 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro86 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o66 withColumn srow_index ro86 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro87 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro88 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro89 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ylo90 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o90 add ro89 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro90 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro91 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro91 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro92 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o88 over ro92 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro93 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o80 withColumn srow_index ro93 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro94 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ylo95 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o95 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro95 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro96 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o87 join ro94 ro96 sinner e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro97 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o97 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro98 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o98 writeTo stest_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro99 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o99 tableProperty sformat-version s1 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro100 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o99 using siceberg e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yro101 (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: c o99 create e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o48 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o49 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o51 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o54 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o55 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o56 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o57 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o58 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o59 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o62 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o64 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o69 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o77 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o83 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o90 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o95 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o100 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o101 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o65 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o66 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o67 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o68 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o70 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o71 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o72 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o73 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o74 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o76 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o78 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o79 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o80 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o81 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o82 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o84 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o85 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o86 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o87 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o88 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o89 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o91 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o92 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o93 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o94 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o96 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Command to send: m d o97 e (clientserver.py:501, send_command) 2025-04-04 18:35:55 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a69ab906-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x226566AA6E28780"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a69ab906-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '65bd0230-f760-4439-a00f-eaab33ad921f' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro' Request method: 'PUT' Request headers: 'Content-Length': '3795' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a69f2784-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1C4D0163CD2C2E0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a69f2784-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'eefb4a61-ef75-45dd-bad5-675b8162ff35' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '5825' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a6a25396-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1ECF9ED1F722080"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a6a25396-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '8d2fd388-3902-4752-8d97-742f8e1e8530' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text' Request method: 'PUT' Request headers: 'Content-Length': '1' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a6a4ffe2-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2120AC7EDE945C0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a6a4ffe2-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'cfcfd5ee-7f00-4faa-a8a6-444f1baccfdf' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '2180' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a6a7ca92-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1F7A4F72BAE68D0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a6a7ca92-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '285521f4-5896-411f-a927-3eef1307d7b2' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro102 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o102 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro103 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ylo104 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o104 add sa e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro104 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro105 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o103 toDF ro105 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro106 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro107 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o107 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro108 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ylo109 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o109 add sb e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro109 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro110 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o108 toDF ro110 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro111 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o111 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro112 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro113 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o113 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro114 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o114 get e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro115 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro116 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yao117 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o116 applyModifiableSettings ro115 ro117 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro118 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o112 cast ro118 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro119 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o111 withColumn sb ro119 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro120 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro121 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro122 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ylo123 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o123 add ro122 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro123 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro124 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro124 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro125 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o121 over ro125 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro126 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o106 withColumn srow_index ro126 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro127 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro128 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro129 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ylo130 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o130 add ro129 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro130 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro131 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro131 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro132 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o128 over ro132 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro133 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o120 withColumn srow_index ro133 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro134 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ylo135 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o135 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro135 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro136 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o127 join ro134 ro136 sinner e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro137 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o137 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro138 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o138 writeTo stest_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro139 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o139 append e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-3-d5a9e94c-6e30-488f-bf96-4d280457c27e-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a6fbaaa4-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-3-d5a9e94c-6e30-488f-bf96-4d280457c27e-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x20034C674754E20"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a6fbaaa4-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '0e2423dc-4798-43f4-aa03-e5e913f1ebf4' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a6fe7e46-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1D6C5CFE5A84FB0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a6fe7e46-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'e441f8e6-89c1-44e8-9e47-4c749f151b69' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v2.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '3230' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a701c542-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v2.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x24C61DAC2FF7CC0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a701c542-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'c2b21a3a-a46c-4a64-8077-e3c163af135e' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro' Request method: 'PUT' Request headers: 'Content-Length': '3795' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a70412ca-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x25B38404CAF3760"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a70412ca-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '953ffd0c-864f-4f3a-94e9-0db6fdc3e2bf' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '5825' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a706786c-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x21DC69B647FBC00"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a706786c-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '2048ed85-4d0a-4ab4-b276-ed78ebb2e51a' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text' Request method: 'PUT' Request headers: 'Content-Length': '1' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a708aa88-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x22603F23E14E600"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a708aa88-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'd3e0393c-f9cc-4681-84e4-1844d9622daf' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-7669783325233029165-1-693836ee-b6f4-465e-9391-e2dbe763e0b7.avro' Request method: 'PUT' Request headers: 'Content-Length': '3865' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a70aac48-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-7669783325233029165-1-693836ee-b6f4-465e-9391-e2dbe763e0b7.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1C24440E5916F30"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a70aac48-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '397d0b72-32b2-4d11-ab82-cd56c74f43c0' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/693836ee-b6f4-465e-9391-e2dbe763e0b7-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '5823' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a70cd8b0-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/693836ee-b6f4-465e-9391-e2dbe763e0b7-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x20918A429CA1B20"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a70cd8b0-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '440e287c-91e6-480e-80ff-7846fa201876' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '2180' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a70efa64-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:58 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:58 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2358BF8B1CFC080"' 'last-modified': 'Fri, 04 Apr 2025 18:35:58 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a70efa64-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '9c896446-7ded-409f-bc39-9b4256db2282' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:58 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:58 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-3-d5a9e94c-6e30-488f-bf96-4d280457c27e-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-7669783325233029165-1-693836ee-b6f4-465e-9391-e2dbe763e0b7.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/693836ee-b6f4-465e-9391-e2dbe763e0b7-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro140 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o140 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro141 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ylo142 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o142 add sa e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro142 e (clientserver.py:501, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Answer received: !yro143 (clientserver.py:512, send_command) 2025-04-04 18:35:58 [ 633 ] DEBUG : Command to send: c o141 toDF ro143 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro144 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro145 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o145 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro146 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ylo147 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o147 add sb e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro147 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro148 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o146 toDF ro148 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro149 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o149 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro150 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro151 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o151 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro152 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o152 get e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro153 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro154 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yao155 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o154 applyModifiableSettings ro153 ro155 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro156 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o150 cast ro156 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro157 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o149 withColumn sb ro157 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro158 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro159 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro160 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ylo161 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o161 add ro160 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro161 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro162 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro162 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro163 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o159 over ro163 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro164 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o144 withColumn srow_index ro164 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro165 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro166 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro167 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ylo168 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o168 add ro167 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro168 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro169 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro169 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro170 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o166 over ro170 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro171 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o158 withColumn srow_index ro171 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro172 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ylo173 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o173 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro173 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro174 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o165 join ro172 ro174 sinner e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro175 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o175 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro176 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o176 writeTo stest_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yro177 (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: c o177 append e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o104 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o109 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o117 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o123 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o102 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o103 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o105 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o106 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o107 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o108 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o110 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o111 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o112 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o113 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o114 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o116 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o118 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o119 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o121 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o122 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o124 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o125 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o126 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o130 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o135 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o142 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o147 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o155 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o128 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o129 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o131 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o132 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o133 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o134 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o136 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o137 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o138 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o139 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o140 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o141 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o143 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o145 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o146 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o148 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o149 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o150 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o151 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o152 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o154 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o156 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o157 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o161 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o168 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Command to send: m d o173 e (clientserver.py:501, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-3-d5a9e94c-6e30-488f-bf96-4d280457c27e-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a750a446-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-3-d5a9e94c-6e30-488f-bf96-4d280457c27e-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1D3E91823431940"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a750a446-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '548f45d3-e12e-4069-8ddc-1959502a90ff' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a75294c2-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1F644E1617CD0E0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a75294c2-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'f7412777-9319-47c7-87c3-dc38eab74374' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-5-15a21675-db0d-4cd0-8439-17ace0571b66-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a7548ba6-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-5-15a21675-db0d-4cd0-8439-17ace0571b66-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x259E1AE8CA58040"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a7548ba6-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '78a97fe7-5d1f-425b-9022-7d4c56c4911d' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v2.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '3230' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a755fb62-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v2.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2600EE4AED11420"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a755fb62-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'eaccce30-2757-47aa-b525-9cba41892f00' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro' Request method: 'PUT' Request headers: 'Content-Length': '3795' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a7574b70-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1C050AC55E4AB70"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a7574b70-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '1255da8f-adbd-440b-9678-abade25a7448' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '5825' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a758949e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1D73CC284F315A0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a758949e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '16b433b0-abaf-43e6-a36e-ce771f9af1aa' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/0c8bb84c-70a5-43c9-a86c-605b21ede418-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '5823' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a759b8ce-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/0c8bb84c-70a5-43c9-a86c-605b21ede418-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1BD65AE7E542910"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a759b8ce-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'c1891767-e8b3-4bc4-b471-13f2a3540c8e' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text' Request method: 'PUT' Request headers: 'Content-Length': '1' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a75ad98e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2122428156388A0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a75ad98e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '22709612-27a8-4ba3-b676-70153b8b1718' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-7669783325233029165-1-693836ee-b6f4-465e-9391-e2dbe763e0b7.avro' Request method: 'PUT' Request headers: 'Content-Length': '3865' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a75c56f6-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-7669783325233029165-1-693836ee-b6f4-465e-9391-e2dbe763e0b7.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1C17C6BE0BC7530"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a75c56f6-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '59276973-60cc-4bb9-96f6-bc6022a0ebf0' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/693836ee-b6f4-465e-9391-e2dbe763e0b7-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '5823' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a75d6762-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/693836ee-b6f4-465e-9391-e2dbe763e0b7-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x24F253269CF21A0"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a75d6762-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '30cea335-8e15-4fae-a6e6-596d0e14f30d' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-4273467181642946234-1-0c8bb84c-70a5-43c9-a86c-605b21ede418.avro' Request method: 'PUT' Request headers: 'Content-Length': '3902' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a75e7f9e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-4273467181642946234-1-0c8bb84c-70a5-43c9-a86c-605b21ede418.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1FEB68E80536480"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a75e7f9e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '8eb40c19-7793-498d-83f5-d60bc35bf8bf' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '2180' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a75fb198-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1D2BBC296E69750"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a75fb198-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '44ddb224-b7c9-4d99-8cf9-0cb2d37c74c2' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v3.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '4281' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a760c498-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:35:59 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v3.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:35:59 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1DAA83705FBF910"' 'last-modified': 'Fri, 04 Apr 2025 18:35:59 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a760c498-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '6ff62887-61ac-4964-8ae8-811704c4d5b1' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:35:59 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:35:59 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-3-d5a9e94c-6e30-488f-bf96-4d280457c27e-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-5-15a21675-db0d-4cd0-8439-17ace0571b66-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/0c8bb84c-70a5-43c9-a86c-605b21ede418-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-7669783325233029165-1-693836ee-b6f4-465e-9391-e2dbe763e0b7.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/693836ee-b6f4-465e-9391-e2dbe763e0b7-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-4273467181642946234-1-0c8bb84c-70a5-43c9-a86c-605b21ede418.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v3.metadata.json'] (test.py:645, add_df) 2025-04-04 18:35:59 [ 633 ] INFO : Setup complete. files: ['/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-3-d5a9e94c-6e30-488f-bf96-4d280457c27e-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-1-fdcebcbc-e4c0-4d25-9c2f-66b73e178b87-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/data/00000-5-15a21675-db0d-4cd0-8439-17ace0571b66-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-8546313456554855404-1-9eff5e87-4578-429a-82db-3c9a91e459e8.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/9eff5e87-4578-429a-82db-3c9a91e459e8-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/0c8bb84c-70a5-43c9-a86c-605b21ede418-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-7669783325233029165-1-693836ee-b6f4-465e-9391-e2dbe763e0b7.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/693836ee-b6f4-465e-9391-e2dbe763e0b7-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/snap-4273467181642946234-1-0c8bb84c-70a5-43c9-a86c-605b21ede418.avro', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/metadata/v3.metadata.json'] (test.py:653, test_cluster_table_function) 2025-04-04 18:35:59 [ 633 ] DEBUG : Executing query SELECT * FROM system.clusters on node1 (cluster.py:3677, query) 2025-04-04 18:35:59 [ 633 ] INFO : Clusters setup: cluster_simple 1 1 0 1 node1 172.16.1.9 9000 1 default 0 0 0 \N \N \N cluster_simple 1 1 0 2 node2 172.16.1.8 9000 0 default 0 0 0 \N \N \N cluster_simple 1 1 0 3 node3 172.16.1.10 9000 0 default 0 0 0 \N \N \N (test.py:657, test_cluster_table_function) 2025-04-04 18:35:59 [ 633 ] DEBUG : Executing query SELECT * FROM icebergAzure(azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/', format=Parquet) on node1 (cluster.py:3677, query) 2025-04-04 18:35:59 [ 633 ] DEBUG : Executing query SELECT * FROM icebergAzureCluster('cluster_simple', azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/', format=Parquet) on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query SELECT * FROM icebergAzure(azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/', format=Parquet) SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8; CREATE TABLE test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 ENGINE=IcebergAzure(azure, container = mycontainer, storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/', format=Parquet) SETTINGS object_storage_cluster = 'cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o159 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o160 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o162 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o163 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o164 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o165 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o166 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o167 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o169 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o170 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o171 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o172 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o174 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o175 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o176 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Command to send: m d o177 e (clientserver.py:501, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query SELECT * FROM remote('node2', icebergAzureCluster('cluster_simple', azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/', format=Parquet) ) on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS `test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8` SYNC on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8; CREATE TABLE test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 ENGINE=IcebergAzure(azure, container = mycontainer, storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8/', format=Parquet) on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 on node1 (cluster.py:3677, query) 2025-04-04 18:36:00 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_azure_c0476a1c_d8d3_4358_9fea_6164c99defc8 SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) _____________________ test_cluster_table_function[azure-2] _____________________ started_cluster = format_version = '2', storage_type = 'azure' @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs"]) def test_cluster_table_function(started_cluster, format_version, storage_type): if is_arm() and storage_type == "hdfs": pytest.skip("Disabled test IcebergHDFS for aarch64") instance = started_cluster.instances["node1"] spark = started_cluster.spark_session TABLE_NAME = ( "test_iceberg_cluster_" + format_version + "_" + storage_type + "_" + get_uuid_str() ) def add_df(mode): write_iceberg_from_df( spark, generate_data(spark, 0, 100), TABLE_NAME, mode=mode, format_version=format_version, ) files = default_upload_directory( started_cluster, storage_type, f"/iceberg_data/default/{TABLE_NAME}/", f"/iceberg_data/default/{TABLE_NAME}/", ) logging.info(f"Adding another dataframe. result files: {files}") return files files = add_df(mode="overwrite") for i in range(1, len(started_cluster.instances)): files = add_df(mode="append") logging.info(f"Setup complete. files: {files}") assert len(files) == 5 + 4 * (len(started_cluster.instances) - 1) clusters = instance.query(f"SELECT * FROM system.clusters") logging.info(f"Clusters setup: {clusters}") # Regular Query only node1 table_function_expr = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True ) select_regular = ( instance.query(f"SELECT * FROM {table_function_expr}").strip().split() ) # Cluster Query with node1 as coordinator table_function_expr_cluster = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True, run_on_cluster=True, ) query_id_cluster = str(uuid.uuid4()) select_cluster = ( instance.query( f"SELECT * FROM {table_function_expr_cluster}", query_id=query_id_cluster ) .strip() .split() ) # Cluster Query with node1 as coordinator with alternative syntax query_id_cluster_alt_syntax = str(uuid.uuid4()) select_cluster_alt_syntax = ( instance.query( f""" SELECT * FROM {table_function_expr} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_cluster_alt_syntax, ) .strip() .split() ) create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster, object_storage_cluster='cluster_simple') query_id_cluster_table_engine = str(uuid.uuid4()) select_cluster_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_cluster_table_engine, ) .strip() .split() ) select_remote_cluster = ( instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") .strip() .split() ) instance.query(f"DROP TABLE IF EXISTS `{TABLE_NAME}` SYNC") create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster) query_id_pure_table_engine = str(uuid.uuid4()) select_pure_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_pure_table_engine, ) .strip() .split() ) query_id_pure_table_engine_cluster = str(uuid.uuid4()) select_pure_table_engine_cluster = ( instance.query( f""" SELECT * FROM {TABLE_NAME} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_pure_table_engine_cluster, ) .strip() .split() ) # Simple size check assert len(select_regular) == 600 assert len(select_cluster) == 600 assert len(select_cluster_alt_syntax) == 600 > assert len(select_cluster_table_engine) == 600 E AssertionError: assert 1800 == 600 E + where 1800 = len(['0', '1', '1', '2', '2', '3', ...]) test_storage_iceberg/test.py:747: AssertionError ----------------------------- Captured stdout call ----------------------------- 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:01 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:36:02 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:02 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:02 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:02 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:02 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:02 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} ------------------------------ Captured log call ------------------------------- 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro178 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o178 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro179 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo180 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o180 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro180 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro181 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o179 toDF ro181 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro182 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro183 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o183 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro184 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo185 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o185 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro185 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro186 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o184 toDF ro186 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro187 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o187 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro188 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro189 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o189 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro190 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o190 get e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro191 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro192 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yao193 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o192 applyModifiableSettings ro191 ro193 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro194 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o188 cast ro194 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro195 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o187 withColumn sb ro195 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro196 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro197 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro198 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo199 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o199 add ro198 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro199 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro200 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro200 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro201 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o197 over ro201 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro202 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o182 withColumn srow_index ro202 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro203 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro204 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro205 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo206 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o206 add ro205 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro206 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro207 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro207 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro208 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o204 over ro208 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro209 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o196 withColumn srow_index ro209 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o47 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o26 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o63 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o99 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o98 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o75 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o120 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o127 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o115 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o144 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o158 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o180 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o185 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o193 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o199 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: m d o206 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro210 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo211 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o211 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro211 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro212 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o203 join ro210 ro212 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro213 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o213 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro214 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o214 writeTo stest_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro215 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o215 tableProperty sformat-version s2 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro216 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o215 using siceberg e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro217 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o215 create e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8a953b0-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:01 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:01 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1F406C2784B2450"' 'last-modified': 'Fri, 04 Apr 2025 18:36:01 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8a953b0-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '8bfbe80b-69f8-4f0d-a8f9-c5becd15ddff' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:01 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:01 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro' Request method: 'PUT' Request headers: 'Content-Length': '4292' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8ab53c2-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:01 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:01 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1CEA04337FC48A0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:01 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8ab53c2-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'eb06a226-b760-4722-ad3c-196a66de0b42' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:01 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:01 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text' Request method: 'PUT' Request headers: 'Content-Length': '1' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8ad2828-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:01 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:01 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x220FC23ACC8CA00"' 'last-modified': 'Fri, 04 Apr 2025 18:36:01 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8ad2828-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '84d6df0d-3501-4508-b04b-d0648caddf3f' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:01 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:01 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '6708' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8aead74-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:01 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:01 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x23AC35B9F28F6A0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:01 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8aead74-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '1358ff38-e7e3-499d-8cad-9ece32c6a94a' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:01 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:01 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '1941' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8b035ea-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:01 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:01 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1C71E1154D6C6A0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:01 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8b035ea-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '932d80e5-819f-4ddb-901c-2c10e1e94dc8' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:01 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:01 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro218 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o218 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro219 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo220 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o220 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro220 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro221 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o219 toDF ro221 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro222 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro223 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o223 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro224 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo225 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o225 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro225 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro226 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o224 toDF ro226 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro227 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o227 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro228 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro229 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o229 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro230 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o230 get e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro231 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro232 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yao233 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o232 applyModifiableSettings ro231 ro233 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro234 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o228 cast ro234 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro235 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o227 withColumn sb ro235 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro236 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro237 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro238 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo239 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o239 add ro238 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro239 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro240 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro240 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro241 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o237 over ro241 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro242 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o222 withColumn srow_index ro242 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro243 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro244 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro245 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo246 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o246 add ro245 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro246 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro247 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro247 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro248 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o244 over ro248 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro249 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o236 withColumn srow_index ro249 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro250 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ylo251 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o251 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro251 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro252 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o243 join ro250 ro252 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro253 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o253 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro254 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o254 writeTo stest_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 e (clientserver.py:501, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Answer received: !yro255 (clientserver.py:512, send_command) 2025-04-04 18:36:01 [ 633 ] DEBUG : Command to send: c o255 append e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-9-64f97c3f-94cc-4ea6-b27c-e20737263e48-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8eedb42-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-9-64f97c3f-94cc-4ea6-b27c-e20737263e48-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2423F357C6E90C0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8eedb42-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '83ecfe39-2f92-4398-ab3d-4c6276198a07' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8f0d92e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1D3F75E4255E0C0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8f0d92e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '35f44a79-aa4e-484e-b2e4-7b266c26333f' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/6d8db028-9630-40ad-bc9c-3843c927a0ec-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '6710' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8f37f62-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/6d8db028-9630-40ad-bc9c-3843c927a0ec-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x261CE230BA87100"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8f37f62-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '1d4e40b2-892f-4e64-a2e0-9cc563c212f4' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-8400718670825027267-1-6d8db028-9630-40ad-bc9c-3843c927a0ec.avro' Request method: 'PUT' Request headers: 'Content-Length': '4364' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8f5dbea-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-8400718670825027267-1-6d8db028-9630-40ad-bc9c-3843c927a0ec.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1CE4772C53414D0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8f5dbea-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'c120810d-ba6d-42ea-907d-69b336dfd397' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v2.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '3018' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8f86360-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v2.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x20A1E53928DA460"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8f86360-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '1f4d4a81-cac7-44e5-bbc2-067aba0234eb' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro' Request method: 'PUT' Request headers: 'Content-Length': '4292' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8fa7bfa-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1CB958D38D070F0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8fa7bfa-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '624b0baa-6569-4c30-aefe-954a67919426' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text' Request method: 'PUT' Request headers: 'Content-Length': '1' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8fc2e5a-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1D9AF37CA64E420"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8fc2e5a-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '95a93bbc-2a02-48bf-aa01-9fc7599a831e' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '6708' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8fdc3c8-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1BDC1AB03441D30"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8fdc3c8-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'a77b2e66-bdf8-4f56-802e-746e8dfc2b49' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '1941' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a8ff58f0-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x24E50AD61857F80"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a8ff58f0-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '95293248-fc06-4309-8353-efd04ac7e546' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-9-64f97c3f-94cc-4ea6-b27c-e20737263e48-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/6d8db028-9630-40ad-bc9c-3843c927a0ec-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-8400718670825027267-1-6d8db028-9630-40ad-bc9c-3843c927a0ec.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro256 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o256 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro257 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ylo258 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o258 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro258 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro259 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o257 toDF ro259 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro260 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro261 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o261 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro262 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ylo263 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o263 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro263 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro264 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o262 toDF ro264 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro265 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o265 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro266 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro267 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o267 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro268 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o268 get e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro269 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro270 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yao271 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o270 applyModifiableSettings ro269 ro271 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro272 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o266 cast ro272 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro273 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o265 withColumn sb ro273 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro274 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro275 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro276 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ylo277 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o277 add ro276 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro277 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro278 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro278 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro279 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o275 over ro279 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro280 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o260 withColumn srow_index ro280 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro281 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro282 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro283 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ylo284 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o284 add ro283 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro284 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro285 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro285 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro286 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o282 over ro286 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro287 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o274 withColumn srow_index ro287 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro288 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ylo289 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o289 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro289 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro290 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o281 join ro288 ro290 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o178 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o179 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o181 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o182 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o183 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o184 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o186 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o187 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o188 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o189 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o190 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o192 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o194 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro291 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o291 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o195 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o196 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o197 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o198 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o200 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o201 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o202 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o204 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o205 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o207 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o208 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o209 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o211 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro292 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o292 writeTo stest_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o216 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o217 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o220 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o225 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o233 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o239 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yro293 (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: c o293 append e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o246 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o212 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o213 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o214 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o215 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o218 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o219 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o221 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o222 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o223 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o224 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o226 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o227 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o228 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o229 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o230 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o232 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o234 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o235 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o237 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o238 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o240 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o241 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o242 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o245 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o247 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o251 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o258 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o263 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o271 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o277 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o250 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o252 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o253 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o254 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o255 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o256 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o257 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o259 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o260 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o261 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o262 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o264 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o265 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o266 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o267 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o268 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o270 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o272 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o273 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o275 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o276 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o278 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o279 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o280 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o284 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Command to send: m d o289 e (clientserver.py:501, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-9-64f97c3f-94cc-4ea6-b27c-e20737263e48-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a930d704-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-9-64f97c3f-94cc-4ea6-b27c-e20737263e48-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1DEF01EB714F830"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a930d704-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '001a8942-4b46-4761-bebe-09e686320761' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a932da5e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x250261FCDA861A0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a932da5e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '7a07e6b5-cf48-4944-8b68-a7f5f3108eb0' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-11-9d7ae9b8-3f48-43de-b2c4-74df78c6179e-00001.parquet' Request method: 'PUT' Request headers: 'Content-Length': '967' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a9349556-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-11-9d7ae9b8-3f48-43de-b2c4-74df78c6179e-00001.parquet HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x23B9B0CC0058120"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a9349556-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '81f93150-97b7-432a-b8e6-95226ed15b36' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/6d8db028-9630-40ad-bc9c-3843c927a0ec-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '6710' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a9360080-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/6d8db028-9630-40ad-bc9c-3843c927a0ec-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2642F2630342400"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a9360080-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '18a4d3ef-ab32-449a-8286-318becce91ad' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-8400718670825027267-1-6d8db028-9630-40ad-bc9c-3843c927a0ec.avro' Request method: 'PUT' Request headers: 'Content-Length': '4364' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a937447c-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-8400718670825027267-1-6d8db028-9630-40ad-bc9c-3843c927a0ec.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x21B360416F88240"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a937447c-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '88aef709-4261-42af-bc28-890ab1d1e179' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v2.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '3018' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a938426e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v2.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x206DA8547F5F380"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a938426e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '5b5ff550-edd8-4637-94a7-08ec416340f6' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/35933ba1-5a59-4e2b-bbd4-29eca6d45849-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '6712' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a9394c72-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/35933ba1-5a59-4e2b-bbd4-29eca6d45849-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x1F82D5BF42B26F0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a9394c72-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '4ef7c7ce-23f8-4b51-80ab-8d2bc16688e8' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro' Request method: 'PUT' Request headers: 'Content-Length': '4292' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a93a3e34-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2006C48D0166700"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a93a3e34-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '932a6d99-79bb-44f7-8802-9a83ac1984e6' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text' Request method: 'PUT' Request headers: 'Content-Length': '1' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a93b524c-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x25F177B7B9BDBA0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a93b524c-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '42d866eb-9a50-4e6d-85d8-e2ffb24bf7a1' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro' Request method: 'PUT' Request headers: 'Content-Length': '6708' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a93c9102-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x211D345BAF80960"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a93c9102-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'b29b85b1-374c-4123-84b4-a8e80600fa71' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '1941' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a93df25e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x23ACB49AEC0F8A0"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a93df25e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'be21b76e-66b9-4b33-a7ac-f64b8c0155d1' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v3.metadata.json' Request method: 'PUT' Request headers: 'Content-Length': '4096' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a93f473a-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v3.metadata.json HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x247456E1D312C00"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a93f473a-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': 'e98378eb-17e6-4659-8c78-dbe6558ed62c' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Request URL: 'http://127.0.0.1:30001/devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4879871834339086647-1-35933ba1-5a59-4e2b-bbd4-29eca6d45849.avro' Request method: 'PUT' Request headers: 'Content-Length': '4410' 'x-ms-blob-type': 'REDACTED' 'x-ms-version': 'REDACTED' 'Content-Type': 'application/octet-stream' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': 'a940987e-1183-11f0-b12d-0242ac110002' 'Authorization': 'REDACTED' A body is sent with the request (_universal.py:511, on_request) 2025-04-04 18:36:02 [ 633 ] DEBUG : http://127.0.0.1:30001 "PUT /devstoreaccount1/mycontainer//iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4879871834339086647-1-35933ba1-5a59-4e2b-bbd4-29eca6d45849.avro HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:02 [ 633 ] INFO : Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x25388104286D400"' 'last-modified': 'Fri, 04 Apr 2025 18:36:02 GMT' 'content-md5': 'REDACTED' 'x-ms-client-request-id': 'a940987e-1183-11f0-b12d-0242ac110002' 'x-ms-request-id': '0eb7b8e3-5771-422f-a6ff-76a96bf5bbd3' 'x-ms-version': 'REDACTED' 'date': 'Fri, 04 Apr 2025 18:36:02 GMT' 'x-ms-request-server-encrypted': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' (_universal.py:550, on_response) 2025-04-04 18:36:02 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-9-64f97c3f-94cc-4ea6-b27c-e20737263e48-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-11-9d7ae9b8-3f48-43de-b2c4-74df78c6179e-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/6d8db028-9630-40ad-bc9c-3843c927a0ec-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-8400718670825027267-1-6d8db028-9630-40ad-bc9c-3843c927a0ec.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/35933ba1-5a59-4e2b-bbd4-29eca6d45849-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v3.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4879871834339086647-1-35933ba1-5a59-4e2b-bbd4-29eca6d45849.avro'] (test.py:645, add_df) 2025-04-04 18:36:02 [ 633 ] INFO : Setup complete. files: ['/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-9-64f97c3f-94cc-4ea6-b27c-e20737263e48-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-7-a67d3036-c5ac-4b97-bc7c-8f76aed19e43-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/data/00000-11-9d7ae9b8-3f48-43de-b2c4-74df78c6179e-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/6d8db028-9630-40ad-bc9c-3843c927a0ec-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-8400718670825027267-1-6d8db028-9630-40ad-bc9c-3843c927a0ec.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/35933ba1-5a59-4e2b-bbd4-29eca6d45849-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4384665493055037759-1-3519f9c6-1f63-471b-9933-da9f7eec9602.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/3519f9c6-1f63-471b-9933-da9f7eec9602-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/v3.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/metadata/snap-4879871834339086647-1-35933ba1-5a59-4e2b-bbd4-29eca6d45849.avro'] (test.py:653, test_cluster_table_function) 2025-04-04 18:36:02 [ 633 ] DEBUG : Executing query SELECT * FROM system.clusters on node1 (cluster.py:3677, query) 2025-04-04 18:36:02 [ 633 ] INFO : Clusters setup: cluster_simple 1 1 0 1 node1 172.16.1.9 9000 1 default 0 0 0 \N \N \N cluster_simple 1 1 0 2 node2 172.16.1.8 9000 0 default 0 0 0 \N \N \N cluster_simple 1 1 0 3 node3 172.16.1.10 9000 0 default 0 0 0 \N \N \N (test.py:657, test_cluster_table_function) 2025-04-04 18:36:02 [ 633 ] DEBUG : Executing query SELECT * FROM icebergAzure(azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/', format=Parquet) on node1 (cluster.py:3677, query) 2025-04-04 18:36:02 [ 633 ] DEBUG : Executing query SELECT * FROM icebergAzureCluster('cluster_simple', azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/', format=Parquet) on node1 (cluster.py:3677, query) 2025-04-04 18:36:02 [ 633 ] DEBUG : Executing query SELECT * FROM icebergAzure(azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/', format=Parquet) SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:03 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09; CREATE TABLE test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 ENGINE=IcebergAzure(azure, container = mycontainer, storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/', format=Parquet) SETTINGS object_storage_cluster = 'cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:03 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 on node1 (cluster.py:3677, query) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o285 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o286 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o287 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o288 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o290 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o291 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o292 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Command to send: m d o293 e (clientserver.py:501, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:03 [ 633 ] DEBUG : Executing query SELECT * FROM remote('node2', icebergAzureCluster('cluster_simple', azure, container = 'mycontainer', storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/', format=Parquet) ) on node1 (cluster.py:3677, query) 2025-04-04 18:36:03 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS `test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09` SYNC on node1 (cluster.py:3677, query) 2025-04-04 18:36:03 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09; CREATE TABLE test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 ENGINE=IcebergAzure(azure, container = mycontainer, storage_account_url = 'http://azurite1:30001/devstoreaccount1', blob_path = '/iceberg_data/default/test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09/', format=Parquet) on node1 (cluster.py:3677, query) 2025-04-04 18:36:03 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 on node1 (cluster.py:3677, query) 2025-04-04 18:36:04 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_azure_7ee9b457_efb3_4eac_aecb_cd36cb95de09 SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) _____________________ test_cluster_table_function[hdfs-1] ______________________ started_cluster = format_version = '1', storage_type = 'hdfs' @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs"]) def test_cluster_table_function(started_cluster, format_version, storage_type): if is_arm() and storage_type == "hdfs": pytest.skip("Disabled test IcebergHDFS for aarch64") instance = started_cluster.instances["node1"] spark = started_cluster.spark_session TABLE_NAME = ( "test_iceberg_cluster_" + format_version + "_" + storage_type + "_" + get_uuid_str() ) def add_df(mode): write_iceberg_from_df( spark, generate_data(spark, 0, 100), TABLE_NAME, mode=mode, format_version=format_version, ) files = default_upload_directory( started_cluster, storage_type, f"/iceberg_data/default/{TABLE_NAME}/", f"/iceberg_data/default/{TABLE_NAME}/", ) logging.info(f"Adding another dataframe. result files: {files}") return files files = add_df(mode="overwrite") for i in range(1, len(started_cluster.instances)): files = add_df(mode="append") logging.info(f"Setup complete. files: {files}") assert len(files) == 5 + 4 * (len(started_cluster.instances) - 1) clusters = instance.query(f"SELECT * FROM system.clusters") logging.info(f"Clusters setup: {clusters}") # Regular Query only node1 table_function_expr = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True ) select_regular = ( instance.query(f"SELECT * FROM {table_function_expr}").strip().split() ) # Cluster Query with node1 as coordinator table_function_expr_cluster = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True, run_on_cluster=True, ) query_id_cluster = str(uuid.uuid4()) select_cluster = ( instance.query( f"SELECT * FROM {table_function_expr_cluster}", query_id=query_id_cluster ) .strip() .split() ) # Cluster Query with node1 as coordinator with alternative syntax query_id_cluster_alt_syntax = str(uuid.uuid4()) select_cluster_alt_syntax = ( instance.query( f""" SELECT * FROM {table_function_expr} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_cluster_alt_syntax, ) .strip() .split() ) create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster, object_storage_cluster='cluster_simple') query_id_cluster_table_engine = str(uuid.uuid4()) select_cluster_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_cluster_table_engine, ) .strip() .split() ) select_remote_cluster = ( instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") .strip() .split() ) instance.query(f"DROP TABLE IF EXISTS `{TABLE_NAME}` SYNC") create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster) query_id_pure_table_engine = str(uuid.uuid4()) select_pure_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_pure_table_engine, ) .strip() .split() ) query_id_pure_table_engine_cluster = str(uuid.uuid4()) select_pure_table_engine_cluster = ( instance.query( f""" SELECT * FROM {TABLE_NAME} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_pure_table_engine_cluster, ) .strip() .split() ) # Simple size check assert len(select_regular) == 600 assert len(select_cluster) == 600 assert len(select_cluster_alt_syntax) == 600 > assert len(select_cluster_table_engine) == 600 E AssertionError: assert 1800 == 600 E + where 1800 = len(['0', '1', '1', '2', '2', '3', ...]) test_storage_iceberg/test.py:747: AssertionError ----------------------------- Captured stdout call ----------------------------- 25/04/04 18:36:04 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:04 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:04 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:04 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:04 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:04 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:05 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} ------------------------------ Captured log call ------------------------------- 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro294 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o294 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro295 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ylo296 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o296 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro296 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro297 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o295 toDF ro297 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro298 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro299 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o299 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro300 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ylo301 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o301 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro301 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro302 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o300 toDF ro302 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro303 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o303 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro304 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro305 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o305 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro306 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o306 get e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro307 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro308 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yao309 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o308 applyModifiableSettings ro307 ro309 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro310 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o304 cast ro310 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro311 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o303 withColumn sb ro311 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro312 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro313 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro314 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ylo315 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o315 add ro314 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro315 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro316 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro316 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro317 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o313 over ro317 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro318 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o298 withColumn srow_index ro318 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro319 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro320 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro321 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ylo322 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o322 add ro321 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro322 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro323 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro323 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro324 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o320 over ro324 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro325 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o312 withColumn srow_index ro325 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro326 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o203 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ylo327 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o327 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o210 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro327 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o191 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro328 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o319 join ro326 ro328 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o248 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o249 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o236 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o243 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o244 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o231 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o274 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o281 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o283 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o282 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o296 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o301 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o309 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o315 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o322 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o327 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o294 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o295 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o297 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o298 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o299 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o300 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o302 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o303 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro329 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o329 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o304 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o305 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o306 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o308 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o310 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o311 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o312 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o313 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o314 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o316 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro330 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o330 writeTo stest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o317 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o318 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o320 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o321 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o323 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro331 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o331 tableProperty sformat-version s1 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o324 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro332 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o331 using siceberg e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o325 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro333 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o331 create e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o332 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: m d o333 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 404 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] INFO : MKDIRS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data?user.name=root&op=MKDIRS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fdata%2F00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:04 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 404 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] INFO : MKDIRS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=MKDIRS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0c\x16avro.schema\xfa\x1b{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"added_snapshot_id","type":["null","long"],"doc":"Snapshot ID that added the manifest","default":null,"field-id":503},{"name":"added_data_files_count","type":["null","int"],"doc":"Added entry count","default":null,"field-id":504},{"name":"existing_data_files_count","type":["null","int"],"doc":"Existing entry count","default":null,"field-id":505},{"name":"deleted_data_files_count","type":["null","int"],"doc":"Deleted entry count","default":null,"field-id":506},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507},{"name":"added_rows_count","type":["null","long"],"doc":"Added rows count","default":null,"field-id":512},{"name":"existing_rows_count","type":["null","long"],"doc":"Existing rows count","default":null,"field-id":513},{"name":"deleted_rows_count","type":["null","long"],"doc":"Deleted rows count","default":null,"field-id":514}]}\x14avro.codec\x0edeflate\x16snapshot-id&1537590172061655805\x1cformat-version\x021\x1ciceberg.schema\xb4\x1a{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":503,"name":"added_snapshot_id","required":false,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":false,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":false,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":false,"type":"int","doc":"Deleted entry count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"},{"id":512,"name":"added_rows_count","required":false,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":false,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":false,"type":"long","doc":"Deleted rows count"}]}$parent-snapshot-id\x08null\x006\x8a\xd5\x7f\x1c\x05\x0f\x19V\x19\xb6\x98\xd4<\xbel\x02\xac\x025\x8cK\n\xc20\x14\x00k\x8e#\xc4\xa4\xf9\xf45Wq\xf3\xc8\xe7E\x85\x16\xa1M{\x05\x17^\xc0Kx\x0fw\xee\xbc\x86{Q\x10\x0b2\x8bY\x0c\xcc\x85\x89C\xa4@\xc3\x0e\x93/^$\xca~\xea\x8a(4\x16\xfc\x97\xd8Mc\xa1\x01k\xdc\xa7<\xa2\nY\xc6&\x1aT \x01\x8d\x92\x16\xbd\x83\x88VB\xf6Z\x07\x17\xad\x15=\x15\xbf\x1c\xa1\xb6\xadt\x19x\x93\xda\x96\x9b\x104\x0f\xe42WF\x13\x80RdS\xe2\xbd\xdc\xf8y8\xbe\xb7\x15{\x9d\x9f\x9f\xeb\xe9\xfeX3\xc6\xaa\x85\xdb\xea\xa7/6\x8a\xd5\x7f\x1c\x05\x0f\x19V\x19\xb6\x98\xd4<\xbel', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fsnap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:04 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fversion-hint.text&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:04 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x96.{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"block_size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x021"partition-spec-id\x020\x1ciceberg.schema\xea${"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x00{\x8c\x9f\x1e\xb9;N\x98\xe0\xdf\xd2F\xb2\xf8\x9e\xe0\x02\xa2\x035\x8c\xbbN\xc30\x18\x85\x1d\xd7\x03S\xe9s \x99\xf8\x12\xd7x\x83\x81\x1d\x10\x9d\x7f\xfd\xf1\xa5\xad\xd4\x01\x12g\xcf\xc6\x80\xc4#\xf0\x00H\xf0\x1c\xd9\xd8x\rv\xa4JM*z\xa6\xef\xe8|:\x94\xfe\xbd\xfe\xee?_\xbe\x7f.\xbeh\xb9\xf5\xb1\x8e\xcd\x1a\x02f,CL\xd8\xedr\x99c\x9b\xe1\xb4\xf8]\xd7\xe6\xd8\x80\x84MH-\xa8:\t\xbf\xf4\x15(+,TJ\x18@g=\x18a\x13j];oLy|\x13S\xb8\xd4\\\xa0A\xed\xbc\xe2!]\xd5\xbc\x12Qs\x94f\xc9u\xb26\x08\x94\n\x93\xe1\x93,/\x9f\xb0y\xeeb\x9e\xdf\xdd<\xdc\xafn\x1f\x87\xe2\xed\xbc\xef\xfbk\xca\xe8\xc7\x8c\xbd\xcf\xc8\x08C\xc1\x86b\x02\xc2\x08\xa1\x13,\xc8\x7f\x18\x95\xc7\xeeO\x9d97:\xf4l\xf4\x0e{\x8c\x9f\x1e\xb9;N\x98\xe0\xdf\xd2F\xb2\xf8\x9e\xe0', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2F715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:04 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:04 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 1,\n "table-uuid" : "769b77b5-093d-42a2-b2f0-c3b96a5d3815",\n "location" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55",\n "last-updated-ms" : 1743791764615,\n "last-column-id" : 2,\n "schema" : {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n },\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "partition-spec" : [ ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 1537590172061655805,\n "refs" : {\n "main" : {\n "snapshot-id" : 1537590172061655805,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791764615,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791764615,\n "snapshot-id" : 1537590172061655805\n } ],\n "metadata-log" : [ ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:04 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fv1.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:04 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:04 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:04 GMT, Fri, 04 Apr 2025 18:36:04 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:04 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro334 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o334 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro335 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ylo336 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o336 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro336 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro337 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o335 toDF ro337 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro338 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro339 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o339 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro340 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ylo341 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o341 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro341 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro342 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o340 toDF ro342 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro343 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o343 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro344 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro345 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o345 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro346 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o346 get e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro347 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro348 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yao349 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o348 applyModifiableSettings ro347 ro349 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro350 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o344 cast ro350 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro351 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c o343 withColumn sb ro351 e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro352 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Answer received: !yro353 (clientserver.py:512, send_command) 2025-04-04 18:36:04 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro354 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo355 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o355 add ro354 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro355 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro356 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro356 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro357 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o353 over ro357 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro358 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o338 withColumn srow_index ro358 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro359 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro360 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro361 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo362 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o362 add ro361 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro362 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro363 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro363 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro364 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o360 over ro364 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro365 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o352 withColumn srow_index ro365 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro366 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo367 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o367 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro367 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro368 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o359 join ro366 ro368 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro369 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o369 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro370 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o370 writeTo stest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro371 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o371 append e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fdata%2F00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o336 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o341 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o349 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o329 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o330 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o331 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o334 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o335 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o337 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o339 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o340 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o342 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o343 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o344 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o345 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o346 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o348 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o350 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o351 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o355 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o362 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: m d o367 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fdata%2F00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 1,\n "table-uuid" : "769b77b5-093d-42a2-b2f0-c3b96a5d3815",\n "location" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55",\n "last-updated-ms" : 1743791765286,\n "last-column-id" : 2,\n "schema" : {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n },\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "partition-spec" : [ ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 7927398034163426173,\n "refs" : {\n "main" : {\n "snapshot-id" : 7927398034163426173,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791764615,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro",\n "schema-id" : 0\n }, {\n "snapshot-id" : 7927398034163426173,\n "parent-snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791765286,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "200",\n "total-files-size" : "1934",\n "total-data-files" : "2",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791764615,\n "snapshot-id" : 1537590172061655805\n }, {\n "timestamp-ms" : 1743791765286,\n "snapshot-id" : 7927398034163426173\n } ],\n "metadata-log" : [ {\n "timestamp-ms" : 1743791764615,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json"\n } ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fv2.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x96.{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"block_size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x021"partition-spec-id\x020\x1ciceberg.schema\xea${"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x00\xd4\x8a\xce:\x9a\xb1\xe8\x17{#\xd3\xa5\x83x<\x0e\x02\xa2\x035\x8c1N\xc40\x10E\x1d\xaf\x0b*\xd8\x8b\x988\x8e\'&\x1d\x14\xf4\x80\xa0\x1eM&6\xac\xb4\x05$N\x1f\x89\x82\n\x8e\x80\xa8)h\xb9B\xae\x80D\xc7-\xa8HV\xec\xaf\xde\xd7\x7f\xfaR\xfe>?}~\xbd\xfd<~g\x1f2\xdfphBw\x8b-%\xca\xdb\x10i\xd8\xa6<\x85>\xe1~\xe1\xed\xd0\xa7\xd0a\x81wm\xec\xd16\xd1p\xc5\x0e\xad7\x1e\x9d5\x80T{F0>RY65\x03\xe4\xbb7\xb3D\x17\xa0\xa3\xab\xacc\x0b\xdaX\x1b\xb4#\x1b5y\x06]\x96-D0T\x9d\xc4\xa8\x17\xb98\xbe\xa7\xeea\x08\xe9\xf0\xe2\xec\xea\xf2\xe6\xfcz\xca^\x8e\xc6q<\x95J\xbe\xaf\xd4\xebJ\xcc0ej\xca\x16\x10J\x08\xb9\xc0Z\xfcG\xc9b\xd7y\xdfU]\xcf\x8e<\x98\xbd?\xd4\x8a\xce:\x9a\xb1\xe8\x17{#\xd3\xa5\x83x<\x0e', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Feb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0c\x16avro.schema\xfa\x1b{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"added_snapshot_id","type":["null","long"],"doc":"Snapshot ID that added the manifest","default":null,"field-id":503},{"name":"added_data_files_count","type":["null","int"],"doc":"Added entry count","default":null,"field-id":504},{"name":"existing_data_files_count","type":["null","int"],"doc":"Existing entry count","default":null,"field-id":505},{"name":"deleted_data_files_count","type":["null","int"],"doc":"Deleted entry count","default":null,"field-id":506},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507},{"name":"added_rows_count","type":["null","long"],"doc":"Added rows count","default":null,"field-id":512},{"name":"existing_rows_count","type":["null","long"],"doc":"Existing rows count","default":null,"field-id":513},{"name":"deleted_rows_count","type":["null","long"],"doc":"Deleted rows count","default":null,"field-id":514}]}\x14avro.codec\x0edeflate\x16snapshot-id&7927398034163426173\x1cformat-version\x021\x1ciceberg.schema\xb4\x1a{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":503,"name":"added_snapshot_id","required":false,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":false,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":false,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":false,"type":"int","doc":"Deleted entry count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"},{"id":512,"name":"added_rows_count","required":false,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":false,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":false,"type":"long","doc":"Deleted rows count"}]}$parent-snapshot-id&1537590172061655805\x00\xf9,\xff\x04\x96\x03\xb5\xce1J\x04A\x10\x05\xd0\xdd>\x8e\xd0NOw\xd7\xd4\xf4UL\x8a\xaa\xeej\x15v\x11fz\xf7\x02\x06\x1b\x88\xf9\x86^\xc0\xd8+\x98\x99\t\x82\x99\x070\x17\x05u\xc1@s\xf9\xc1\x0f>|\xde\xdet\xe7YE\xa7S*\xdc\xb8+Zy\xb3j]\xd3\xb9\xd1\xcf\x92W\x9b\xb9\xe9D=\x9d\x95:\x93\x97\xea\xf2\x90#ytH\xd1; N\x98\t\x1cV\x0eAR\x06\xe8\xd6\xda\xf8\xf0\xa8\x12a\x08\xd9[Aa\x1b\x93\xef-W\x8dV4\x84\x01!A\xd2d\xd7\xee\x98\xb7\xd3\xc5\xfb\xc9\xc2\xbc]\xef\xee\x9en^.\x9f\x97\xc6\x98\xc5!\xf7\xcb\xef\xda\xff;\x15{\x18]\xaah\x872\x8e6\x8a\x84/e\xaa\xd6\xc7\xa0\x88\xde+\x94\xf2\x8bz\xf5\xfaq\xbb{x<\xfa#\xfd\x04\xf9,\xff', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fsnap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0c\x16avro.schema\xfa\x1b{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"added_snapshot_id","type":["null","long"],"doc":"Snapshot ID that added the manifest","default":null,"field-id":503},{"name":"added_data_files_count","type":["null","int"],"doc":"Added entry count","default":null,"field-id":504},{"name":"existing_data_files_count","type":["null","int"],"doc":"Existing entry count","default":null,"field-id":505},{"name":"deleted_data_files_count","type":["null","int"],"doc":"Deleted entry count","default":null,"field-id":506},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507},{"name":"added_rows_count","type":["null","long"],"doc":"Added rows count","default":null,"field-id":512},{"name":"existing_rows_count","type":["null","long"],"doc":"Existing rows count","default":null,"field-id":513},{"name":"deleted_rows_count","type":["null","long"],"doc":"Deleted rows count","default":null,"field-id":514}]}\x14avro.codec\x0edeflate\x16snapshot-id&1537590172061655805\x1cformat-version\x021\x1ciceberg.schema\xb4\x1a{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":503,"name":"added_snapshot_id","required":false,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":false,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":false,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":false,"type":"int","doc":"Deleted entry count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"},{"id":512,"name":"added_rows_count","required":false,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":false,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":false,"type":"long","doc":"Deleted rows count"}]}$parent-snapshot-id\x08null\x006\x8a\xd5\x7f\x1c\x05\x0f\x19V\x19\xb6\x98\xd4<\xbel\x02\xac\x025\x8cK\n\xc20\x14\x00k\x8e#\xc4\xa4\xf9\xf45Wq\xf3\xc8\xe7E\x85\x16\xa1M{\x05\x17^\xc0Kx\x0fw\xee\xbc\x86{Q\x10\x0b2\x8bY\x0c\xcc\x85\x89C\xa4@\xc3\x0e\x93/^$\xca~\xea\x8a(4\x16\xfc\x97\xd8Mc\xa1\x01k\xdc\xa7<\xa2\nY\xc6&\x1aT \x01\x8d\x92\x16\xbd\x83\x88VB\xf6Z\x07\x17\xad\x15=\x15\xbf\x1c\xa1\xb6\xadt\x19x\x93\xda\x96\x9b\x104\x0f\xe42WF\x13\x80RdS\xe2\xbd\xdc\xf8y8\xbe\xb7\x15{\x9d\x9f\x9f\xeb\xe9\xfeX3\xc6\xaa\x85\xdb\xea\xa7/6\x8a\xd5\x7f\x1c\x05\x0f\x19V\x19\xb6\x98\xd4<\xbel', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fsnap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'2', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fversion-hint.text&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x96.{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"block_size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x021"partition-spec-id\x020\x1ciceberg.schema\xea${"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x00{\x8c\x9f\x1e\xb9;N\x98\xe0\xdf\xd2F\xb2\xf8\x9e\xe0\x02\xa2\x035\x8c\xbbN\xc30\x18\x85\x1d\xd7\x03S\xe9s \x99\xf8\x12\xd7x\x83\x81\x1d\x10\x9d\x7f\xfd\xf1\xa5\xad\xd4\x01\x12g\xcf\xc6\x80\xc4#\xf0\x00H\xf0\x1c\xd9\xd8x\rv\xa4JM*z\xa6\xef\xe8|:\x94\xfe\xbd\xfe\xee?_\xbe\x7f.\xbeh\xb9\xf5\xb1\x8e\xcd\x1a\x02f,CL\xd8\xedr\x99c\x9b\xe1\xb4\xf8]\xd7\xe6\xd8\x80\x84MH-\xa8:\t\xbf\xf4\x15(+,TJ\x18@g=\x18a\x13j];oLy|\x13S\xb8\xd4\\\xa0A\xed\xbc\xe2!]\xd5\xbc\x12Qs\x94f\xc9u\xb26\x08\x94\n\x93\xe1\x93,/\x9f\xb0y\xeeb\x9e\xdf\xdd<\xdc\xafn\x1f\x87\xe2\xed\xbc\xef\xfbk\xca\xe8\xc7\x8c\xbd\xcf\xc8\x08C\xc1\x86b\x02\xc2\x08\xa1\x13,\xc8\x7f\x18\x95\xc7\xeeO\x9d97:\xf4l\xf4\x0e{\x8c\x9f\x1e\xb9;N\x98\xe0\xdf\xd2F\xb2\xf8\x9e\xe0', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2F715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:05 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 1,\n "table-uuid" : "769b77b5-093d-42a2-b2f0-c3b96a5d3815",\n "location" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55",\n "last-updated-ms" : 1743791764615,\n "last-column-id" : 2,\n "schema" : {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n },\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "partition-spec" : [ ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 1537590172061655805,\n "refs" : {\n "main" : {\n "snapshot-id" : 1537590172061655805,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791764615,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791764615,\n "snapshot-id" : 1537590172061655805\n } ],\n "metadata-log" : [ ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:05 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fv1.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:05 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:05 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:05 GMT, Fri, 04 Apr 2025 18:36:05 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:05 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro372 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o372 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro373 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo374 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o374 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro374 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro375 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o373 toDF ro375 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro376 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro377 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o377 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro378 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo379 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o379 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro379 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro380 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o378 toDF ro380 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro381 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o381 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro382 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro383 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o383 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro384 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o384 get e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro385 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro386 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yao387 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o386 applyModifiableSettings ro385 ro387 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro388 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o382 cast ro388 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro389 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o381 withColumn sb ro389 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro390 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro391 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro392 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo393 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o393 add ro392 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro393 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro394 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro394 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro395 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o391 over ro395 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro396 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o376 withColumn srow_index ro396 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro397 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro398 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro399 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo400 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o400 add ro399 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro400 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro401 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro401 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro402 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o398 over ro402 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro403 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o390 withColumn srow_index ro403 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro404 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ylo405 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o405 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro405 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro406 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o397 join ro404 ro406 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro407 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o407 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro408 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o408 writeTo stest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yro409 (clientserver.py:512, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Command to send: c o409 append e (clientserver.py:501, send_command) 2025-04-04 18:36:05 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fdata%2F00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fdata%2F00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fdata%2F00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 1,\n "table-uuid" : "769b77b5-093d-42a2-b2f0-c3b96a5d3815",\n "location" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55",\n "last-updated-ms" : 1743791765286,\n "last-column-id" : 2,\n "schema" : {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n },\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "partition-spec" : [ ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 7927398034163426173,\n "refs" : {\n "main" : {\n "snapshot-id" : 7927398034163426173,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791764615,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro",\n "schema-id" : 0\n }, {\n "snapshot-id" : 7927398034163426173,\n "parent-snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791765286,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "200",\n "total-files-size" : "1934",\n "total-data-files" : "2",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791764615,\n "snapshot-id" : 1537590172061655805\n }, {\n "timestamp-ms" : 1743791765286,\n "snapshot-id" : 7927398034163426173\n } ],\n "metadata-log" : [ {\n "timestamp-ms" : 1743791764615,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json"\n } ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fv2.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0c\x16avro.schema\xfa\x1b{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"added_snapshot_id","type":["null","long"],"doc":"Snapshot ID that added the manifest","default":null,"field-id":503},{"name":"added_data_files_count","type":["null","int"],"doc":"Added entry count","default":null,"field-id":504},{"name":"existing_data_files_count","type":["null","int"],"doc":"Existing entry count","default":null,"field-id":505},{"name":"deleted_data_files_count","type":["null","int"],"doc":"Deleted entry count","default":null,"field-id":506},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507},{"name":"added_rows_count","type":["null","long"],"doc":"Added rows count","default":null,"field-id":512},{"name":"existing_rows_count","type":["null","long"],"doc":"Existing rows count","default":null,"field-id":513},{"name":"deleted_rows_count","type":["null","long"],"doc":"Deleted rows count","default":null,"field-id":514}]}\x14avro.codec\x0edeflate\x16snapshot-id&6818206824545359152\x1cformat-version\x021\x1ciceberg.schema\xb4\x1a{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":503,"name":"added_snapshot_id","required":false,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":false,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":false,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":false,"type":"int","doc":"Deleted entry count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"},{"id":512,"name":"added_rows_count","required":false,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":false,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":false,"type":"long","doc":"Deleted rows count"}]}$parent-snapshot-id&7927398034163426173\x00zA)\xfd\x97\xea\x84M\x8d\xa1[\xc1\xed\x88Di\x06\xea\x03\xbd\xd0\xbdJ\x03A\x14\x05\xe0d\x1fG\xb8\xee\xec\x9d\x9f\xbb\xf3*6\xc3\xfc\xdcQ!A\xd8\x9d\xf8\x02\x16)\xc4\xc2.\x9d\xbe\x80\xb5\x95\xd6v6"\x08)D\x1f@\xeb\xa0\xe0&\x92B{\xe5\x16\xa78p\xf8\xb8\x8b\xaa>\x8c\x1c\xb8\xdbw\xc9\x17_\'\xce~6)u\xe1\xbe\xb8m\x13\'\xb3\xbep\xe7\x1aw\x90r\xef0d\x11MT\x0eI\x90S(\xb4\xf3\x96\xa2\xd3\x82\xb2\x972\xd8\xa8u=\xe5\xe2\xbf\x17[\xc4l\x14A\x93\xb1\x01\xd5\xa2\x00\xaf\xa4\x04C\xda\x1aM\x92\x08\x13L\xc5\xae?\xee\x8eV{\xa3\xea\xf9a\xf9\xf2~{~q3\xae\xaaj\xb4\xb9\xbb\xf1:\x16\x7fN\xe5\xa0\xb4\x91\x11!P\xf0\xa0\xec\xe0\xf5\x99\x15\x04\x96r\xcd\xd5\x96\xed\x96\xfa1PWg\xf3\xeb\xa7\xcb\xd7\x93\xe5\xffS\xa9\xd1\xad\xb0\x99\xc0\xa4\xb6\x05\x15\x82\x1c\x946\x03*\xc9\xc3G\x91uJ?\xa8\xa7o\x9fW\xf3\xfb\xc7\x9d_\xd2/zA)\xfd\x97\xea\x84M\x8d\xa1[\xc1\xed\x88Di', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fsnap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x96.{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"block_size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x021"partition-spec-id\x020\x1ciceberg.schema\xea${"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x00\xd4\x8a\xce:\x9a\xb1\xe8\x17{#\xd3\xa5\x83x<\x0e\x02\xa2\x035\x8c1N\xc40\x10E\x1d\xaf\x0b*\xd8\x8b\x988\x8e\'&\x1d\x14\xf4\x80\xa0\x1eM&6\xac\xb4\x05$N\x1f\x89\x82\n\x8e\x80\xa8)h\xb9B\xae\x80D\xc7-\xa8HV\xec\xaf\xde\xd7\x7f\xfaR\xfe>?}~\xbd\xfd<~g\x1f2\xdfphBw\x8b-%\xca\xdb\x10i\xd8\xa6<\x85>\xe1~\xe1\xed\xd0\xa7\xd0a\x81wm\xec\xd16\xd1p\xc5\x0e\xad7\x1e\x9d5\x80T{F0>RY65\x03\xe4\xbb7\xb3D\x17\xa0\xa3\xab\xacc\x0b\xdaX\x1b\xb4#\x1b5y\x06]\x96-D0T\x9d\xc4\xa8\x17\xb98\xbe\xa7\xeea\x08\xe9\xf0\xe2\xec\xea\xf2\xe6\xfcz\xca^\x8e\xc6q<\x95J\xbe\xaf\xd4\xebJ\xcc0ej\xca\x16\x10J\x08\xb9\xc0Z\xfcG\xc9b\xd7y\xdfU]\xcf\x8e<\x98\xbd?\xd4\x8a\xce:\x9a\xb1\xe8\x17{#\xd3\xa5\x83x<\x0e', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Feb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0c\x16avro.schema\xfa\x1b{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"added_snapshot_id","type":["null","long"],"doc":"Snapshot ID that added the manifest","default":null,"field-id":503},{"name":"added_data_files_count","type":["null","int"],"doc":"Added entry count","default":null,"field-id":504},{"name":"existing_data_files_count","type":["null","int"],"doc":"Existing entry count","default":null,"field-id":505},{"name":"deleted_data_files_count","type":["null","int"],"doc":"Deleted entry count","default":null,"field-id":506},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507},{"name":"added_rows_count","type":["null","long"],"doc":"Added rows count","default":null,"field-id":512},{"name":"existing_rows_count","type":["null","long"],"doc":"Existing rows count","default":null,"field-id":513},{"name":"deleted_rows_count","type":["null","long"],"doc":"Deleted rows count","default":null,"field-id":514}]}\x14avro.codec\x0edeflate\x16snapshot-id&7927398034163426173\x1cformat-version\x021\x1ciceberg.schema\xb4\x1a{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":503,"name":"added_snapshot_id","required":false,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":false,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":false,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":false,"type":"int","doc":"Deleted entry count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"},{"id":512,"name":"added_rows_count","required":false,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":false,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":false,"type":"long","doc":"Deleted rows count"}]}$parent-snapshot-id&1537590172061655805\x00\xf9,\xff\x04\x96\x03\xb5\xce1J\x04A\x10\x05\xd0\xdd>\x8e\xd0NOw\xd7\xd4\xf4UL\x8a\xaa\xeej\x15v\x11fz\xf7\x02\x06\x1b\x88\xf9\x86^\xc0\xd8+\x98\x99\t\x82\x99\x070\x17\x05u\xc1@s\xf9\xc1\x0f>|\xde\xdet\xe7YE\xa7S*\xdc\xb8+Zy\xb3j]\xd3\xb9\xd1\xcf\x92W\x9b\xb9\xe9D=\x9d\x95:\x93\x97\xea\xf2\x90#ytH\xd1; N\x98\t\x1cV\x0eAR\x06\xe8\xd6\xda\xf8\xf0\xa8\x12a\x08\xd9[Aa\x1b\x93\xef-W\x8dV4\x84\x01!A\xd2d\xd7\xee\x98\xb7\xd3\xc5\xfb\xc9\xc2\xbc]\xef\xee\x9en^.\x9f\x97\xc6\x98\xc5!\xf7\xcb\xef\xda\xff;\x15{\x18]\xaah\x872\x8e6\x8a\x84/e\xaa\xd6\xc7\xa0\x88\xde+\x94\xf2\x8bz\xf5\xfaq\xbb{x<\xfa#\xfd\x04\xf9,\xff', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fsnap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0c\x16avro.schema\xfa\x1b{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"added_snapshot_id","type":["null","long"],"doc":"Snapshot ID that added the manifest","default":null,"field-id":503},{"name":"added_data_files_count","type":["null","int"],"doc":"Added entry count","default":null,"field-id":504},{"name":"existing_data_files_count","type":["null","int"],"doc":"Existing entry count","default":null,"field-id":505},{"name":"deleted_data_files_count","type":["null","int"],"doc":"Deleted entry count","default":null,"field-id":506},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507},{"name":"added_rows_count","type":["null","long"],"doc":"Added rows count","default":null,"field-id":512},{"name":"existing_rows_count","type":["null","long"],"doc":"Existing rows count","default":null,"field-id":513},{"name":"deleted_rows_count","type":["null","long"],"doc":"Deleted rows count","default":null,"field-id":514}]}\x14avro.codec\x0edeflate\x16snapshot-id&1537590172061655805\x1cformat-version\x021\x1ciceberg.schema\xb4\x1a{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":503,"name":"added_snapshot_id","required":false,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":false,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":false,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":false,"type":"int","doc":"Deleted entry count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"},{"id":512,"name":"added_rows_count","required":false,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":false,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":false,"type":"long","doc":"Deleted rows count"}]}$parent-snapshot-id\x08null\x006\x8a\xd5\x7f\x1c\x05\x0f\x19V\x19\xb6\x98\xd4<\xbel\x02\xac\x025\x8cK\n\xc20\x14\x00k\x8e#\xc4\xa4\xf9\xf45Wq\xf3\xc8\xe7E\x85\x16\xa1M{\x05\x17^\xc0Kx\x0fw\xee\xbc\x86{Q\x10\x0b2\x8bY\x0c\xcc\x85\x89C\xa4@\xc3\x0e\x93/^$\xca~\xea\x8a(4\x16\xfc\x97\xd8Mc\xa1\x01k\xdc\xa7<\xa2\nY\xc6&\x1aT \x01\x8d\x92\x16\xbd\x83\x88VB\xf6Z\x07\x17\xad\x15=\x15\xbf\x1c\xa1\xb6\xadt\x19x\x93\xda\x96\x9b\x104\x0f\xe42WF\x13\x80RdS\xe2\xbd\xdc\xf8y8\xbe\xb7\x15{\x9d\x9f\x9f\xeb\xe9\xfeX3\xc6\xaa\x85\xdb\xea\xa7/6\x8a\xd5\x7f\x1c\x05\x0f\x19V\x19\xb6\x98\xd4<\xbel', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fsnap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'3', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o354 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o356 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o357 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o358 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o359 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o360 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o361 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o363 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o364 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o365 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o366 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o368 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o369 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o370 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o371 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o374 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o379 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o387 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o393 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o400 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o372 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o373 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o375 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fversion-hint.text&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o376 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o377 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o378 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o380 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o381 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o382 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o383 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o384 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o386 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o388 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o389 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o390 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o391 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o392 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o394 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o395 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o396 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o398 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o399 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o401 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o402 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o403 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Command to send: m d o405 e (clientserver.py:501, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x96.{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"block_size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x021"partition-spec-id\x020\x1ciceberg.schema\xea${"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x00\xeaT\x8d\x99\xde\xdc\x03f\xcf\x82T\x87\\\xb5\x80O\x02\xa0\x035\x8c=N\xc40\x14\x84\x1d\xaf\x0b*\xe0"&\xfe\xc5\xb8\x83\x82\x1e\x10\xd4O\xcf/\x0e m\x01\x89\xd3\xe7\x04H\\\x00z\nn\x00u\x0e@I\x818\x01G Y\xedN\xf5\x8d\xe6\xd3p\xfe\xf3\xf5\xfd\xfb\xf7\xf9\xfc\xfaQ\xbd\xf3\xfa\x9er\xca\xdd-4X\xb0nr\x8b\xc3\xba\xd4%\xf7\x05v\x0b\xad\x87\xbe\xe4\x0e4\xdc5m\x0f&\xb5\x8a\x8e\xc9\x81\t*\x803\xca\x03\xc6@\xe0Uh\xd1\xda\x14\xc9\xfbz\xf3\xa6\x96H\x1d\xa45Q\x19\xb2^&MN:\xb4J\x9e8t\xd2FlR\xc8!9\x0cr\x91\xf5\xd1\x03v\x8fC.\xfb\x17gW\x977\xe7\xd7S\xf5t0\x8e\xe3)\x17\xfcm%^Vl\x86\xa9\x12S\xb5\x00\x13\x8c\xf1\x05\x0e\xd96\x82\xebM\xa7]\x171\xce\x0e\xdf\x9b\xbd\x7f\xeaT\x8d\x99\xde\xdc\x03f\xcf\x82T\x87\\\xb5\x80O', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fd822f647-1f21-4820-a433-67596573772d-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x96.{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"block_size_in_bytes","type":"long","field-id":105},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x021"partition-spec-id\x020\x1ciceberg.schema\xea${"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]}},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":105,"name":"block_size_in_bytes","required":true,"type":"long"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x00{\x8c\x9f\x1e\xb9;N\x98\xe0\xdf\xd2F\xb2\xf8\x9e\xe0\x02\xa2\x035\x8c\xbbN\xc30\x18\x85\x1d\xd7\x03S\xe9s \x99\xf8\x12\xd7x\x83\x81\x1d\x10\x9d\x7f\xfd\xf1\xa5\xad\xd4\x01\x12g\xcf\xc6\x80\xc4#\xf0\x00H\xf0\x1c\xd9\xd8x\rv\xa4JM*z\xa6\xef\xe8|:\x94\xfe\xbd\xfe\xee?_\xbe\x7f.\xbeh\xb9\xf5\xb1\x8e\xcd\x1a\x02f,CL\xd8\xedr\x99c\x9b\xe1\xb4\xf8]\xd7\xe6\xd8\x80\x84MH-\xa8:\t\xbf\xf4\x15(+,TJ\x18@g=\x18a\x13j];oLy|\x13S\xb8\xd4\\\xa0A\xed\xbc\xe2!]\xd5\xbc\x12Qs\x94f\xc9u\xb26\x08\x94\n\x93\xe1\x93,/\x9f\xb0y\xeeb\x9e\xdf\xdd<\xdc\xafn\x1f\x87\xe2\xed\xbc\xef\xfbk\xca\xe8\xc7\x8c\xbd\xcf\xc8\x08C\xc1\x86b\x02\xc2\x08\xa1\x13,\xc8\x7f\x18\x95\xc7\xeeO\x9d97:\xf4l\xf4\x0e{\x8c\x9f\x1e\xb9;N\x98\xe0\xdf\xd2F\xb2\xf8\x9e\xe0', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2F715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 1,\n "table-uuid" : "769b77b5-093d-42a2-b2f0-c3b96a5d3815",\n "location" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55",\n "last-updated-ms" : 1743791764615,\n "last-column-id" : 2,\n "schema" : {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n },\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "partition-spec" : [ ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 1537590172061655805,\n "refs" : {\n "main" : {\n "snapshot-id" : 1537590172061655805,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791764615,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791764615,\n "snapshot-id" : 1537590172061655805\n } ],\n "metadata-log" : [ ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fv1.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:06 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 1,\n "table-uuid" : "769b77b5-093d-42a2-b2f0-c3b96a5d3815",\n "location" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55",\n "last-updated-ms" : 1743791765976,\n "last-column-id" : 2,\n "schema" : {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n },\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "partition-spec" : [ ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 6818206824545359152,\n "refs" : {\n "main" : {\n "snapshot-id" : 6818206824545359152,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791764615,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro",\n "schema-id" : 0\n }, {\n "snapshot-id" : 7927398034163426173,\n "parent-snapshot-id" : 1537590172061655805,\n "timestamp-ms" : 1743791765286,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "200",\n "total-files-size" : "1934",\n "total-data-files" : "2",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro",\n "schema-id" : 0\n }, {\n "snapshot-id" : 6818206824545359152,\n "parent-snapshot-id" : 7927398034163426173,\n "timestamp-ms" : 1743791765976,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "300",\n "total-files-size" : "2901",\n "total-data-files" : "3",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791764615,\n "snapshot-id" : 1537590172061655805\n }, {\n "timestamp-ms" : 1743791765286,\n "snapshot-id" : 7927398034163426173\n }, {\n "timestamp-ms" : 1743791765976,\n "snapshot-id" : 6818206824545359152\n } ],\n "metadata-log" : [ {\n "timestamp-ms" : 1743791764615,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json"\n }, {\n "timestamp-ms" : 1743791765286,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json"\n } ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:06 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55%2Fmetadata%2Fv3.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:06 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:06 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:06 GMT, Fri, 04 Apr 2025 18:36:06 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:06 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:06 [ 633 ] INFO : Setup complete. files: ['/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-13-0a5a39c2-df8b-40e3-a156-3f77d0a12af5-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-17-32902c35-b1c4-4a30-84a4-39adb7e7b4a7-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/data/00000-15-f4624c25-022e-4a2f-a7c5-33d5f50a68ff-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-6818206824545359152-1-d822f647-1f21-4820-a433-67596573772d.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/eb4563c2-b7ba-4921-afe4-be33675959e9-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-7927398034163426173-1-eb4563c2-b7ba-4921-afe4-be33675959e9.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/snap-1537590172061655805-1-715809f7-6d88-4bb3-be9f-243e7722e5dd.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/d822f647-1f21-4820-a433-67596573772d-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/715809f7-6d88-4bb3-be9f-243e7722e5dd-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/metadata/v3.metadata.json'] (test.py:653, test_cluster_table_function) 2025-04-04 18:36:06 [ 633 ] DEBUG : Executing query SELECT * FROM system.clusters on node1 (cluster.py:3677, query) 2025-04-04 18:36:06 [ 633 ] INFO : Clusters setup: cluster_simple 1 1 0 1 node1 172.16.1.9 9000 1 default 0 0 0 \N \N \N cluster_simple 1 1 0 2 node2 172.16.1.8 9000 0 default 0 0 0 \N \N \N cluster_simple 1 1 0 3 node3 172.16.1.10 9000 0 default 0 0 0 \N \N \N (test.py:657, test_cluster_table_function) 2025-04-04 18:36:06 [ 633 ] DEBUG : Executing query SELECT * FROM icebergHDFS(hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/', format=Parquet, url = 'hdfs://hdfs1:9000/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:06 [ 633 ] DEBUG : Executing query SELECT * FROM icebergHDFSCluster('cluster_simple', hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/', format=Parquet, url = 'hdfs://hdfs1:9000/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:06 [ 633 ] DEBUG : Executing query SELECT * FROM icebergHDFS(hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/', format=Parquet, url = 'hdfs://hdfs1:9000/') SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:07 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55; CREATE TABLE test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 ENGINE=IcebergHDFS(hdfs, filename = 'iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/', format=Parquet, url = 'hdfs://hdfs1:9000/') SETTINGS object_storage_cluster = 'cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:07 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 on node1 (cluster.py:3677, query) 2025-04-04 18:36:07 [ 633 ] DEBUG : Executing query SELECT * FROM remote('node2', icebergHDFSCluster('cluster_simple', hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/', format=Parquet, url = 'hdfs://hdfs1:9000/') ) on node1 (cluster.py:3677, query) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: m d o406 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: m d o407 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: m d o408 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: m d o409 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS `test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55` SYNC on node1 (cluster.py:3677, query) 2025-04-04 18:36:07 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55; CREATE TABLE test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 ENGINE=IcebergHDFS(hdfs, filename = 'iceberg_data/default/test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55/', format=Parquet, url = 'hdfs://hdfs1:9000/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:07 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 on node1 (cluster.py:3677, query) 2025-04-04 18:36:07 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_hdfs_2bf0c6c4_2707_4205_a97c_507fa33b9c55 SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) _____________________ test_cluster_table_function[hdfs-2] ______________________ started_cluster = format_version = '2', storage_type = 'hdfs' @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs"]) def test_cluster_table_function(started_cluster, format_version, storage_type): if is_arm() and storage_type == "hdfs": pytest.skip("Disabled test IcebergHDFS for aarch64") instance = started_cluster.instances["node1"] spark = started_cluster.spark_session TABLE_NAME = ( "test_iceberg_cluster_" + format_version + "_" + storage_type + "_" + get_uuid_str() ) def add_df(mode): write_iceberg_from_df( spark, generate_data(spark, 0, 100), TABLE_NAME, mode=mode, format_version=format_version, ) files = default_upload_directory( started_cluster, storage_type, f"/iceberg_data/default/{TABLE_NAME}/", f"/iceberg_data/default/{TABLE_NAME}/", ) logging.info(f"Adding another dataframe. result files: {files}") return files files = add_df(mode="overwrite") for i in range(1, len(started_cluster.instances)): files = add_df(mode="append") logging.info(f"Setup complete. files: {files}") assert len(files) == 5 + 4 * (len(started_cluster.instances) - 1) clusters = instance.query(f"SELECT * FROM system.clusters") logging.info(f"Clusters setup: {clusters}") # Regular Query only node1 table_function_expr = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True ) select_regular = ( instance.query(f"SELECT * FROM {table_function_expr}").strip().split() ) # Cluster Query with node1 as coordinator table_function_expr_cluster = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True, run_on_cluster=True, ) query_id_cluster = str(uuid.uuid4()) select_cluster = ( instance.query( f"SELECT * FROM {table_function_expr_cluster}", query_id=query_id_cluster ) .strip() .split() ) # Cluster Query with node1 as coordinator with alternative syntax query_id_cluster_alt_syntax = str(uuid.uuid4()) select_cluster_alt_syntax = ( instance.query( f""" SELECT * FROM {table_function_expr} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_cluster_alt_syntax, ) .strip() .split() ) create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster, object_storage_cluster='cluster_simple') query_id_cluster_table_engine = str(uuid.uuid4()) select_cluster_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_cluster_table_engine, ) .strip() .split() ) select_remote_cluster = ( instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") .strip() .split() ) instance.query(f"DROP TABLE IF EXISTS `{TABLE_NAME}` SYNC") create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster) query_id_pure_table_engine = str(uuid.uuid4()) select_pure_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_pure_table_engine, ) .strip() .split() ) query_id_pure_table_engine_cluster = str(uuid.uuid4()) select_pure_table_engine_cluster = ( instance.query( f""" SELECT * FROM {TABLE_NAME} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_pure_table_engine_cluster, ) .strip() .split() ) # Simple size check assert len(select_regular) == 600 assert len(select_cluster) == 600 assert len(select_cluster_alt_syntax) == 600 > assert len(select_cluster_table_engine) == 600 E AssertionError: assert 1800 == 600 E + where 1800 = len(['0', '1', '1', '2', '2', '3', ...]) test_storage_iceberg/test.py:747: AssertionError ----------------------------- Captured stdout call ----------------------------- 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} 25/04/04 18:36:09 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:09 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:09 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:09 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:09 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:09 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} ------------------------------ Captured log call ------------------------------- 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro410 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o410 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro411 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ylo412 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o412 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro412 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro413 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o411 toDF ro413 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro414 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro415 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o415 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro416 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ylo417 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o417 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro417 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro418 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o416 toDF ro418 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro419 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o419 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro420 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro421 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o421 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro422 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o422 get e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro423 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro424 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yao425 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o424 applyModifiableSettings ro423 ro425 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro426 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o420 cast ro426 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro427 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c o419 withColumn sb ro427 e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro428 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro429 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yro430 (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:07 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo431 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o431 add ro430 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro431 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro432 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro432 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro433 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o429 over ro433 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro434 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o414 withColumn srow_index ro434 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro435 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro436 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro437 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo438 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o438 add ro437 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro438 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro439 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro439 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro440 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o436 over ro440 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro441 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o428 withColumn srow_index ro441 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro442 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo443 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o443 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro443 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro444 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o435 join ro442 ro444 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro445 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o445 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro446 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o446 writeTo stest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro447 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o447 tableProperty sformat-version s2 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro448 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o447 using siceberg e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro449 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o447 create e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 404 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] INFO : MKDIRS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data?user.name=root&op=MKDIRS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fdata%2F00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 404 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] INFO : MKDIRS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=MKDIRS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x10\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x925{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"sequence_number","type":["null","long"],"default":null,"field-id":3},{"name":"file_sequence_number","type":["null","long"],"default":null,"field-id":4},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"content","type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes","field-id":134},{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"doc":"Partition data tuple, schema based on the partition spec","field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"equality_ids","type":["null",{"type":"array","items":"int","element-id":136}],"doc":"Equality comparison field IDs","default":null,"field-id":135},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x022"partition-spec-id\x020\x1ciceberg.schema\xca+{"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":3,"name":"sequence_number","required":false,"type":"long"},{"id":4,"name":"file_sequence_number","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":134,"name":"content","required":true,"type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes"},{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]},"doc":"Partition data tuple, schema based on the partition spec"},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":135,"name":"equality_ids","required":false,"type":{"type":"list","element-id":136,"element":"int","element-required":true},"doc":"Equality comparison field IDs"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x0econtent\x08data\x00h\xff\xed\x8bA \'I\x080\xe5W\xdcr\xf1-\x02\x9c\x035\x8c1N\x031\x10E\xbd\x8e\x0b*\xe0"f\xd7\xbbv\x12\xd3\xa5\xa0\x87\x08\xea\xd1x=C\x90R$\xbb\xde\x0b \xd1\xe6\x08\x1c\x80\x82\x863 -E\xee\xc1)\x10kD~\xf5\xfe\xcc\xd3\x97\xf2\xfb\xe3\xf9\xe7\xf3\xeb\xe5x-\x84x\x97\xe5SK\x81\xbaG\x88\x98\xb0\x8c\xc48lS\x99\xa8Op\xfa\xb4\xdb\xa1O\xd4A\r\x9b\xc8=\x18C\xce\xbb\xc8\xc0\xb5Y\x80E\xaa\x009Dh\x88\x17\x8c\xb6\xf1\x9e\xa7\xa1\xbcV\xe5h\xe3\xf5\x12\xe3\x1c\x97m\xa5}\x13jmi\xba\x06W;m#;\xac\x82EK\x8d\xce\xb2\xb9\xdaa\xb7\x1f(\x9d\xdf\xae\xd6w\x0f7\xf7cq\xb8\x90J\xbe\xcd\xd4\xebLL0\x16j,2\x08%\x84\xccp)\xfe\xa3\xa4\xf9\xeb\xed\xa9+\xef\'G\x9ee\xf1\x17h\xff\xed\x8bA \'I\x080\xe5W\xdcr\xf1-', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2F1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x16avro.schema\x9e\x1e{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"content","type":"int","doc":"Contents of the manifest: 0=data, 1=deletes","field-id":517},{"name":"sequence_number","type":"long","doc":"Sequence number when the manifest was added","field-id":515},{"name":"min_sequence_number","type":"long","doc":"Lowest sequence number in the manifest","field-id":516},{"name":"added_snapshot_id","type":"long","doc":"Snapshot ID that added the manifest","field-id":503},{"name":"added_data_files_count","type":"int","doc":"Added entry count","field-id":504},{"name":"existing_data_files_count","type":"int","doc":"Existing entry count","field-id":505},{"name":"deleted_data_files_count","type":"int","doc":"Deleted entry count","field-id":506},{"name":"added_rows_count","type":"long","doc":"Added rows count","field-id":512},{"name":"existing_rows_count","type":"long","doc":"Existing rows count","field-id":513},{"name":"deleted_rows_count","type":"long","doc":"Deleted rows count","field-id":514},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507}]}\x14avro.codec\x0edeflate\x16snapshot-id&2111356058946522356\x1cformat-version\x022\x1esequence-number\x021\x1ciceberg.schema\xd6\x1f{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":517,"name":"content","required":true,"type":"int","doc":"Contents of the manifest: 0=data, 1=deletes"},{"id":515,"name":"sequence_number","required":true,"type":"long","doc":"Sequence number when the manifest was added"},{"id":516,"name":"min_sequence_number","required":true,"type":"long","doc":"Lowest sequence number in the manifest"},{"id":503,"name":"added_snapshot_id","required":true,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":true,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":true,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":true,"type":"int","doc":"Deleted entry count"},{"id":512,"name":"added_rows_count","required":true,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":true,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":true,"type":"long","doc":"Deleted rows count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"}]}$parent-snapshot-id\x08null\x00\x81\x1a\x91\xc6h\x8exe`\x10\xec\xd8\xe0\xb4:a\x02\xa4\x025\xcc\xd1\t\xc20\x10\x80\xe1\x98}b\x9a\xf6j9\x979.\xc9\x9d\x15*B\x9b:\x80\xe0\x0e\xee\xe1\x0c\x82>\xb8\x87\x0f\xce \x8a\xe0\xeb\xff\xc1\x7f\xb6~\x9b$\xca\xb8\xa1\xcc\x85}\x16\xe5y(\xbe\xc8T\xe8/i\x98\xa7"#\xd5\xd4g\x9d(\x04i\xb1\xcdJZ\x87\x8e\x80\xa5"\xd6\x98\xa9\x11\xed\x94\xa1AT\xf6;)\xfc;\x86\x1c\xa5\xe9b\xeb$ap\x00\x91\x1d\xc2*9\x8d\nXC\xaa\x12\xa2\xdbUK>\x8c\xfbWo\x8c\xb5\xcf\xcb\xf1}\xbd\x9f\x1ekk\xccm\xf1-\xe6\x03\x81\x1a\x91\xc6h\x8exe`\x10\xec\xd8\xe0\xb4:a', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fsnap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fversion-hint.text&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 2,\n "table-uuid" : "13d813c2-9ac5-40bf-9809-0177cfcab88e",\n "location" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa",\n "last-sequence-number" : 1,\n "last-updated-ms" : 1743791768254,\n "last-column-id" : 2,\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 2111356058946522356,\n "refs" : {\n "main" : {\n "snapshot-id" : 2111356058946522356,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "sequence-number" : 1,\n "snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768254,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791768254,\n "snapshot-id" : 2111356058946522356\n } ],\n "metadata-log" : [ ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o326 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o328 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o319 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o307 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o338 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o352 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o353 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o347 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o397 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o404 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o412 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o417 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o425 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o431 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o438 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o443 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o410 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o411 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o413 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o414 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o415 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o416 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o418 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o419 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o420 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o421 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o422 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o424 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o426 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o427 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o428 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o429 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o430 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o432 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o433 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o434 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o435 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o436 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o437 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o439 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o440 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o441 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o442 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fv1.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o444 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o445 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro450 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o450 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o448 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: m d o449 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro451 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo452 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o452 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro452 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro453 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o451 toDF ro453 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro454 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro455 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o455 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro456 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo457 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o457 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro457 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro458 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o456 toDF ro458 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro459 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o459 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro460 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro461 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o461 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro462 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o462 get e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro463 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro464 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yao465 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o464 applyModifiableSettings ro463 ro465 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro466 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o460 cast ro466 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro467 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o459 withColumn sb ro467 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro468 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro469 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro470 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo471 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o471 add ro470 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro471 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro472 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro472 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro473 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o469 over ro473 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro474 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o454 withColumn srow_index ro474 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro475 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro476 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro477 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo478 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o478 add ro477 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro478 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro479 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro479 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro480 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o476 over ro480 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro481 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o468 withColumn srow_index ro481 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro482 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ylo483 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o483 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro483 e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro484 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o475 join ro482 ro484 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro485 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o485 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro486 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o486 writeTo stest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yro487 (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Command to send: c o487 append e (clientserver.py:501, send_command) 2025-04-04 18:36:08 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fdata%2F00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fdata%2F00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x10\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x925{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"sequence_number","type":["null","long"],"default":null,"field-id":3},{"name":"file_sequence_number","type":["null","long"],"default":null,"field-id":4},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"content","type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes","field-id":134},{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"doc":"Partition data tuple, schema based on the partition spec","field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"equality_ids","type":["null",{"type":"array","items":"int","element-id":136}],"doc":"Equality comparison field IDs","default":null,"field-id":135},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x022"partition-spec-id\x020\x1ciceberg.schema\xca+{"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":3,"name":"sequence_number","required":false,"type":"long"},{"id":4,"name":"file_sequence_number","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":134,"name":"content","required":true,"type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes"},{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]},"doc":"Partition data tuple, schema based on the partition spec"},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":135,"name":"equality_ids","required":false,"type":{"type":"list","element-id":136,"element":"int","element-required":true},"doc":"Equality comparison field IDs"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x0econtent\x08data\x00\x94\xf9\x9d\x0cD\'e\xb9\xc2\x90\xdf.\xdc\x82M\xcf\x02\x9a\x035\x8c1N\x031\x10E\xbd\x8e\x0b*\xc2E\xcc\xc6\x1b{\x8dK\nz@\xa1\x1eM\xec\x99$R\n\xd8\xf5V4\x9c\x80\x8a\x9a\x8a\x8a\x82SP\xac\x94{p\n\x84X#\xf2\xab\xf7\xf5\x9f\xbe\x94\x9fo\xdf?\x87\x97\xaf\xa7G!\xc4\x87\xacw\x91\xd6\xd4m a\xc6:\x11\xe3\xb0\xcfu\xa6>\xc3q\x89\xfb\xa1\xcf\xd4A\x03\xdb\xc4=\x18C.\xb8\xc4\xc0\x8d\xf1`\x91\x16\x80\xbcN\xb0$\xf6\x8cv\x19\x02OG\xe5mQ\xa2\x1b\xa3\x13\xa3i\x93o\xb5k\x9b\xa8m\xbc \x8d.\x04\xed\xd1\x92\xb7\xcep\x98\xbc"\x9b\xf3{\xec\x1e\x06\xca\xa7\xd7\x97\xb77wW\xab\xb1z\x9eK%\xdfg\xeau&&\x18+5V\x05\x84\x12B\x168\x13\xffQ\xd2\xfc\xf5x\xec*\x84\xc9\x91\'E\xfc\x05\x94\xf9\x9d\x0cD\'e\xb9\xc2\x90\xdf.\xdc\x82M\xcf', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Feb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 2,\n "table-uuid" : "13d813c2-9ac5-40bf-9809-0177cfcab88e",\n "location" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa",\n "last-sequence-number" : 2,\n "last-updated-ms" : 1743791768691,\n "last-column-id" : 2,\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 4431766687501486370,\n "refs" : {\n "main" : {\n "snapshot-id" : 4431766687501486370,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "sequence-number" : 1,\n "snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768254,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro",\n "schema-id" : 0\n }, {\n "sequence-number" : 2,\n "snapshot-id" : 4431766687501486370,\n "parent-snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768691,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "200",\n "total-files-size" : "1934",\n "total-data-files" : "2",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791768254,\n "snapshot-id" : 2111356058946522356\n }, {\n "timestamp-ms" : 1743791768691,\n "snapshot-id" : 4431766687501486370\n } ],\n "metadata-log" : [ {\n "timestamp-ms" : 1743791768254,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json"\n } ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fv2.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x16avro.schema\x9e\x1e{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"content","type":"int","doc":"Contents of the manifest: 0=data, 1=deletes","field-id":517},{"name":"sequence_number","type":"long","doc":"Sequence number when the manifest was added","field-id":515},{"name":"min_sequence_number","type":"long","doc":"Lowest sequence number in the manifest","field-id":516},{"name":"added_snapshot_id","type":"long","doc":"Snapshot ID that added the manifest","field-id":503},{"name":"added_data_files_count","type":"int","doc":"Added entry count","field-id":504},{"name":"existing_data_files_count","type":"int","doc":"Existing entry count","field-id":505},{"name":"deleted_data_files_count","type":"int","doc":"Deleted entry count","field-id":506},{"name":"added_rows_count","type":"long","doc":"Added rows count","field-id":512},{"name":"existing_rows_count","type":"long","doc":"Existing rows count","field-id":513},{"name":"deleted_rows_count","type":"long","doc":"Deleted rows count","field-id":514},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507}]}\x14avro.codec\x0edeflate\x16snapshot-id&4431766687501486370\x1cformat-version\x022\x1esequence-number\x022\x1ciceberg.schema\xd6\x1f{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":517,"name":"content","required":true,"type":"int","doc":"Contents of the manifest: 0=data, 1=deletes"},{"id":515,"name":"sequence_number","required":true,"type":"long","doc":"Sequence number when the manifest was added"},{"id":516,"name":"min_sequence_number","required":true,"type":"long","doc":"Lowest sequence number in the manifest"},{"id":503,"name":"added_snapshot_id","required":true,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":true,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":true,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":true,"type":"int","doc":"Deleted entry count"},{"id":512,"name":"added_rows_count","required":true,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":true,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":true,"type":"long","doc":"Deleted rows count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"}]}$parent-snapshot-id&2111356058946522356\x00\xd1`\xa68\x8aE\xdb\x91h.\xc0\xd1\xdc\xdeH\xc7\x04\x94\x03\xb5\xce\xd1I\x041\x10\x80\xe1\x18\xae\x9d\xb8\xc9\xeedsc3a&\x99\xf1\x84;\x84\xdd\x9c/\xbe\x88`\x07>\\\x01v`\x0b\n\x8a\xd8\x86\x88X\x82\x1c\x8apv\xe0\xf3\x0f?\xdf\xcevgEX\xa6\xd3\\\xa9QWEi\xbbn]\x93\xb9\xe5C)\xeb\xed\xdcd\xca}^U\x9ds\x08\x121V\xcd\xda\x87\x94\x81\xc4gR\xaey\x10MJ0 *u\x1bi\xf4{\x14\x0eX\x8b_:\x9019\xf0Kv8Jt\x10|*cE\xeecr\x1b\x7fL\x17\xd3\xf9\xc7\xca\x98\xc5\xe2\xe1\xeek\xffr\xfbvui\x8dy:2\xc6\x9a\xdd\xbf+Ce\x19\x12G\'\x05\x83\x03`r\x08cq\xca\n\xd8C\xf1\x05\xf1\xa0\xfc\xfcQZ\xfb~\x7f\xbd\x7f|\xbey=\xf9S~\x03\xd1`\xa68\x8aE\xdb\x91h.\xc0\xd1\xdc\xdeH\xc7', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fsnap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x10\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x925{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"sequence_number","type":["null","long"],"default":null,"field-id":3},{"name":"file_sequence_number","type":["null","long"],"default":null,"field-id":4},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"content","type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes","field-id":134},{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"doc":"Partition data tuple, schema based on the partition spec","field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"equality_ids","type":["null",{"type":"array","items":"int","element-id":136}],"doc":"Equality comparison field IDs","default":null,"field-id":135},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x022"partition-spec-id\x020\x1ciceberg.schema\xca+{"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":3,"name":"sequence_number","required":false,"type":"long"},{"id":4,"name":"file_sequence_number","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":134,"name":"content","required":true,"type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes"},{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]},"doc":"Partition data tuple, schema based on the partition spec"},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":135,"name":"equality_ids","required":false,"type":{"type":"list","element-id":136,"element":"int","element-required":true},"doc":"Equality comparison field IDs"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x0econtent\x08data\x00h\xff\xed\x8bA \'I\x080\xe5W\xdcr\xf1-\x02\x9c\x035\x8c1N\x031\x10E\xbd\x8e\x0b*\xe0"f\xd7\xbbv\x12\xd3\xa5\xa0\x87\x08\xea\xd1x=C\x90R$\xbb\xde\x0b \xd1\xe6\x08\x1c\x80\x82\x863 -E\xee\xc1)\x10kD~\xf5\xfe\xcc\xd3\x97\xf2\xfb\xe3\xf9\xe7\xf3\xeb\xe5x-\x84x\x97\xe5SK\x81\xbaG\x88\x98\xb0\x8c\xc48lS\x99\xa8Op\xfa\xb4\xdb\xa1O\xd4A\r\x9b\xc8=\x18C\xce\xbb\xc8\xc0\xb5Y\x80E\xaa\x009Dh\x88\x17\x8c\xb6\xf1\x9e\xa7\xa1\xbcV\xe5h\xe3\xf5\x12\xe3\x1c\x97m\xa5}\x13jmi\xba\x06W;m#;\xac\x82EK\x8d\xce\xb2\xb9\xdaa\xb7\x1f(\x9d\xdf\xae\xd6w\x0f7\xf7cq\xb8\x90J\xbe\xcd\xd4\xebLL0\x16j,2\x08%\x84\xccp)\xfe\xa3\xa4\xf9\xeb\xed\xa9+\xef\'G\x9ee\xf1\x17h\xff\xed\x8bA \'I\x080\xe5W\xdcr\xf1-', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2F1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x16avro.schema\x9e\x1e{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"content","type":"int","doc":"Contents of the manifest: 0=data, 1=deletes","field-id":517},{"name":"sequence_number","type":"long","doc":"Sequence number when the manifest was added","field-id":515},{"name":"min_sequence_number","type":"long","doc":"Lowest sequence number in the manifest","field-id":516},{"name":"added_snapshot_id","type":"long","doc":"Snapshot ID that added the manifest","field-id":503},{"name":"added_data_files_count","type":"int","doc":"Added entry count","field-id":504},{"name":"existing_data_files_count","type":"int","doc":"Existing entry count","field-id":505},{"name":"deleted_data_files_count","type":"int","doc":"Deleted entry count","field-id":506},{"name":"added_rows_count","type":"long","doc":"Added rows count","field-id":512},{"name":"existing_rows_count","type":"long","doc":"Existing rows count","field-id":513},{"name":"deleted_rows_count","type":"long","doc":"Deleted rows count","field-id":514},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507}]}\x14avro.codec\x0edeflate\x16snapshot-id&2111356058946522356\x1cformat-version\x022\x1esequence-number\x021\x1ciceberg.schema\xd6\x1f{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":517,"name":"content","required":true,"type":"int","doc":"Contents of the manifest: 0=data, 1=deletes"},{"id":515,"name":"sequence_number","required":true,"type":"long","doc":"Sequence number when the manifest was added"},{"id":516,"name":"min_sequence_number","required":true,"type":"long","doc":"Lowest sequence number in the manifest"},{"id":503,"name":"added_snapshot_id","required":true,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":true,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":true,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":true,"type":"int","doc":"Deleted entry count"},{"id":512,"name":"added_rows_count","required":true,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":true,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":true,"type":"long","doc":"Deleted rows count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"}]}$parent-snapshot-id\x08null\x00\x81\x1a\x91\xc6h\x8exe`\x10\xec\xd8\xe0\xb4:a\x02\xa4\x025\xcc\xd1\t\xc20\x10\x80\xe1\x98}b\x9a\xf6j9\x979.\xc9\x9d\x15*B\x9b:\x80\xe0\x0e\xee\xe1\x0c\x82>\xb8\x87\x0f\xce \x8a\xe0\xeb\xff\xc1\x7f\xb6~\x9b$\xca\xb8\xa1\xcc\x85}\x16\xe5y(\xbe\xc8T\xe8/i\x98\xa7"#\xd5\xd4g\x9d(\x04i\xb1\xcdJZ\x87\x8e\x80\xa5"\xd6\x98\xa9\x11\xed\x94\xa1AT\xf6;)\xfc;\x86\x1c\xa5\xe9b\xeb$ap\x00\x91\x1d\xc2*9\x8d\nXC\xaa\x12\xa2\xdbUK>\x8c\xfbWo\x8c\xb5\xcf\xcb\xf1}\xbd\x9f\x1ekk\xccm\xf1-\xe6\x03\x81\x1a\x91\xc6h\x8exe`\x10\xec\xd8\xe0\xb4:a', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fsnap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:08 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:08 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:08 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:08 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:08 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:08 GMT, Fri, 04 Apr 2025 18:36:08 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'2', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fversion-hint.text&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 2,\n "table-uuid" : "13d813c2-9ac5-40bf-9809-0177cfcab88e",\n "location" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa",\n "last-sequence-number" : 1,\n "last-updated-ms" : 1743791768254,\n "last-column-id" : 2,\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 2111356058946522356,\n "refs" : {\n "main" : {\n "snapshot-id" : 2111356058946522356,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "sequence-number" : 1,\n "snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768254,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791768254,\n "snapshot-id" : 2111356058946522356\n } ],\n "metadata-log" : [ ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fv1.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro488 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o488 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro489 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ylo490 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o490 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro490 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro491 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o489 toDF ro491 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro492 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro493 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o493 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro494 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ylo495 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o495 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro495 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro496 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o494 toDF ro496 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro497 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o497 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro498 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro499 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o499 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro500 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o500 get e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro501 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro502 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yao503 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o502 applyModifiableSettings ro501 ro503 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro504 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o498 cast ro504 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro505 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o497 withColumn sb ro505 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro506 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro507 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro508 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ylo509 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o509 add ro508 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro509 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro510 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro510 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro511 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o507 over ro511 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro512 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o492 withColumn srow_index ro512 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro513 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro514 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro515 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ylo516 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o516 add ro515 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro516 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro517 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro517 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro518 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o514 over ro518 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro519 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o506 withColumn srow_index ro519 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro520 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ylo521 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o521 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro521 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro522 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o513 join ro520 ro522 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro523 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o523 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro524 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o524 writeTo stest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yro525 (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: c o525 append e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o452 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o457 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o465 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o447 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o450 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o451 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o453 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o455 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o456 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o458 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o459 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o460 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o461 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o462 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o464 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o466 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o467 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o471 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o478 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o483 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o472 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o473 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o474 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o475 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o476 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o477 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o479 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o480 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o481 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o482 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o484 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o485 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o486 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o487 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o490 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o495 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o503 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o509 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o516 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o521 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o488 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o489 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o491 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o492 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o493 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o494 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o496 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o497 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o498 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o499 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o500 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o502 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o504 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o505 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o506 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o507 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o508 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o510 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o511 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o512 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o513 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o514 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o515 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o517 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o518 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o519 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o520 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o522 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o523 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o524 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Command to send: m d o525 e (clientserver.py:501, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fdata%2F00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fdata%2F00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'PAR1\x15\x00\x15\xc0\x0c\x15\xf6\x02\x15\xbd\xf3\xd4\x95\x06\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00-\xc5\xd7"\x02\x00\x00\x00\xc0\x90Y![$+\xb3\xb23C\x8a\xec\x8c\xca\x8e\xf2\xff\xff\xe0\xc1\xdd\xcb\x05\x02\xffZ\xdc\xea6\x07\xdd\xee\x0ew\xba\xcb\xdd\xeeq\xc8aG\xdc\xeb>\xf7;\xea\x01\x0fz\xc8\xc3\x1e\xf1\xa8\xc7<\xee\x98\'<\xe9\xb8\xa7\x9c\xf0\xb4g<\xeb9\xcf;\xe9\x05/z\xc9\xcb^\xf1\xaaSN;\xe35\xaf{\xc3\x9b\xde\xf2\xb6w\x9c\xf5\xae\xf7\xbc\xef\x03\x1f\xfa\xc89\x1f\xfb\xc4\xa7\xce\xfb\xcc\x05\x17}\xee\x0b\x97|\xe9+_\xfb\xc6\xb7\xbe\xf3\xbd\xcb~\xf0\xa3\x9f\xfc\xec\x8a\xab\xae\xf9\xc5\xaf~\xf3\xbb?\xfc\xe9/\xd7\xfd\xed\x1f7\xdc\xf4\xaf\xff\x00\x02\xc2\xe7q \x03\x00\x00\x15\x00\x15\xa0\t\x15\xea\x02\x15\x8d\xbc\xbf\xb8\t\x1c\x15\xc8\x01\x15\x00\x15\x08\x15\x08\x00\x00\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\x1d\xd2\xb1\x81B1\x0cD\xc1\x0f\xd7\x10X\xb2$\xf7\xdf\x18\xe7I^\xb4\x93\xed\xeby\x9e\xef\xeb?\xeb&n\xf2f\xdf\xd4M\xdf\xcc\xcdy\xdf\xf1G\xbf\xba44uki\xeb(\xbb\xd8\xc5.v\xb1\x8b]\xecb\x17\xbb\xd8\xc5\x06\x1bl\xb0\xc1\x06\x1bl\xb0\xc1\x06\x1bl\xb2\xc9&\x9bl\xb2\xc9&\x9bl\xb2\xc9nv\xb3\x9b\xdd\xecf7\xbb\xd9\xcdnv\xb3\xc5\x16[l\xb1\xc5\x16[l\xb1\xc5\x16\xdbl\xb3\xcd6\xdbl\xb3\xcd6\xdbl\xb3\xc3\x0e;\xec\xb0\xc3\x0e;\xec\xb0\xc3\x0e{\xd8\xc3\x1e\xf6\xb0\x87=\xeca\x0f{\xd8s\xfe|\xe3\xf3\x03\xd4\xdb\x86\xadP\x02\x00\x00\x19\x11\x02\x19\x18\x08\x00\x00\x00\x00\x00\x00\x00\x00\x19\x18\x08c\x00\x00\x00\x00\x00\x00\x00\x15\x02\x19\x16\x00\x00\x19\x11\x02\x19\x18\x011\x19\x18\x0299\x15\x02\x19\x16\x00\x00\x19\x1c\x16\x08\x15\xaa\x03\x16\x00\x00\x00\x19\x1c\x16\xb2\x03\x15\x9e\x03\x16\x00\x00\x00\x15\x02\x19\x00&\xb2\x03\x1c\x15\x0c\x19%\x08\x00\x19\x18\x01b\x15\x04\x16\xc8\x01\x16\xd4\t\x16\x9e\x03&\xb2\x03<6\x00(\x0299\x18\x011\x00\x19\x1c\x15\x00\x15\x00\x15\x02\x00\x00\x16\xc8\x07\x15\x18\x16\x8e\x07\x15$\x00\x16\xc8\x16\x16\xc8\x01&\x08\x16\xc8\x06\x14\x00\x00\x19\x1c\x18\x0eiceberg.schema\x18\x90\x01{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":true,"type":"long"},{"id":2,"name":"b","required":true,"type":"string"}]}\x00\x18Jparquet-mr version 1.12.3 (build f8dced182c4c1fbdec6ccb3185537b5a01e6ed6b)\x19,\x1c\x00\x00\x1c\x00\x00\x00\xcf\x01\x00\x00PAR1', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fdata%2F00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x10\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x925{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"sequence_number","type":["null","long"],"default":null,"field-id":3},{"name":"file_sequence_number","type":["null","long"],"default":null,"field-id":4},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"content","type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes","field-id":134},{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"doc":"Partition data tuple, schema based on the partition spec","field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"equality_ids","type":["null",{"type":"array","items":"int","element-id":136}],"doc":"Equality comparison field IDs","default":null,"field-id":135},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x022"partition-spec-id\x020\x1ciceberg.schema\xca+{"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":3,"name":"sequence_number","required":false,"type":"long"},{"id":4,"name":"file_sequence_number","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":134,"name":"content","required":true,"type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes"},{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]},"doc":"Partition data tuple, schema based on the partition spec"},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":135,"name":"equality_ids","required":false,"type":{"type":"list","element-id":136,"element":"int","element-required":true},"doc":"Equality comparison field IDs"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x0econtent\x08data\x00\x94\xf9\x9d\x0cD\'e\xb9\xc2\x90\xdf.\xdc\x82M\xcf\x02\x9a\x035\x8c1N\x031\x10E\xbd\x8e\x0b*\xc2E\xcc\xc6\x1b{\x8dK\nz@\xa1\x1eM\xec\x99$R\n\xd8\xf5V4\x9c\x80\x8a\x9a\x8a\x8a\x82SP\xac\x94{p\n\x84X#\xf2\xab\xf7\xf5\x9f\xbe\x94\x9fo\xdf?\x87\x97\xaf\xa7G!\xc4\x87\xacw\x91\xd6\xd4m a\xc6:\x11\xe3\xb0\xcfu\xa6>\xc3q\x89\xfb\xa1\xcf\xd4A\x03\xdb\xc4=\x18C.\xb8\xc4\xc0\x8d\xf1`\x91\x16\x80\xbcN\xb0$\xf6\x8cv\x19\x02OG\xe5mQ\xa2\x1b\xa3\x13\xa3i\x93o\xb5k\x9b\xa8m\xbc \x8d.\x04\xed\xd1\x92\xb7\xcep\x98\xbc"\x9b\xf3{\xec\x1e\x06\xca\xa7\xd7\x97\xb77wW\xab\xb1z\x9eK%\xdfg\xeau&&\x18+5V\x05\x84\x12B\x168\x13\xffQ\xd2\xfc\xf5x\xec*\x84\xc9\x91\'E\xfc\x05\x94\xf9\x9d\x0cD\'e\xb9\xc2\x90\xdf.\xdc\x82M\xcf', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Feb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 2,\n "table-uuid" : "13d813c2-9ac5-40bf-9809-0177cfcab88e",\n "location" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa",\n "last-sequence-number" : 2,\n "last-updated-ms" : 1743791768691,\n "last-column-id" : 2,\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 4431766687501486370,\n "refs" : {\n "main" : {\n "snapshot-id" : 4431766687501486370,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "sequence-number" : 1,\n "snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768254,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro",\n "schema-id" : 0\n }, {\n "sequence-number" : 2,\n "snapshot-id" : 4431766687501486370,\n "parent-snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768691,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "200",\n "total-files-size" : "1934",\n "total-data-files" : "2",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791768254,\n "snapshot-id" : 2111356058946522356\n }, {\n "timestamp-ms" : 1743791768691,\n "snapshot-id" : 4431766687501486370\n } ],\n "metadata-log" : [ {\n "timestamp-ms" : 1743791768254,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json"\n } ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fv2.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x16avro.schema\x9e\x1e{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"content","type":"int","doc":"Contents of the manifest: 0=data, 1=deletes","field-id":517},{"name":"sequence_number","type":"long","doc":"Sequence number when the manifest was added","field-id":515},{"name":"min_sequence_number","type":"long","doc":"Lowest sequence number in the manifest","field-id":516},{"name":"added_snapshot_id","type":"long","doc":"Snapshot ID that added the manifest","field-id":503},{"name":"added_data_files_count","type":"int","doc":"Added entry count","field-id":504},{"name":"existing_data_files_count","type":"int","doc":"Existing entry count","field-id":505},{"name":"deleted_data_files_count","type":"int","doc":"Deleted entry count","field-id":506},{"name":"added_rows_count","type":"long","doc":"Added rows count","field-id":512},{"name":"existing_rows_count","type":"long","doc":"Existing rows count","field-id":513},{"name":"deleted_rows_count","type":"long","doc":"Deleted rows count","field-id":514},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507}]}\x14avro.codec\x0edeflate\x16snapshot-id&4431766687501486370\x1cformat-version\x022\x1esequence-number\x022\x1ciceberg.schema\xd6\x1f{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":517,"name":"content","required":true,"type":"int","doc":"Contents of the manifest: 0=data, 1=deletes"},{"id":515,"name":"sequence_number","required":true,"type":"long","doc":"Sequence number when the manifest was added"},{"id":516,"name":"min_sequence_number","required":true,"type":"long","doc":"Lowest sequence number in the manifest"},{"id":503,"name":"added_snapshot_id","required":true,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":true,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":true,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":true,"type":"int","doc":"Deleted entry count"},{"id":512,"name":"added_rows_count","required":true,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":true,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":true,"type":"long","doc":"Deleted rows count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"}]}$parent-snapshot-id&2111356058946522356\x00\xd1`\xa68\x8aE\xdb\x91h.\xc0\xd1\xdc\xdeH\xc7\x04\x94\x03\xb5\xce\xd1I\x041\x10\x80\xe1\x18\xae\x9d\xb8\xc9\xeedsc3a&\x99\xf1\x84;\x84\xdd\x9c/\xbe\x88`\x07>\\\x01v`\x0b\n\x8a\xd8\x86\x88X\x82\x1c\x8apv\xe0\xf3\x0f?\xdf\xcevgEX\xa6\xd3\\\xa9QWEi\xbbn]\x93\xb9\xe5C)\xeb\xed\xdcd\xca}^U\x9ds\x08\x121V\xcd\xda\x87\x94\x81\xc4gR\xaey\x10MJ0 *u\x1bi\xf4{\x14\x0eX\x8b_:\x9019\xf0Kv8Jt\x10|*cE\xeecr\x1b\x7fL\x17\xd3\xf9\xc7\xca\x98\xc5\xe2\xe1\xeek\xffr\xfbvui\x8dy:2\xc6\x9a\xdd\xbf+Ce\x19\x12G\'\x05\x83\x03`r\x08cq\xca\n\xd8C\xf1\x05\xf1\xa0\xfc\xfcQZ\xfb~\x7f\xbd\x7f|\xbey=\xf9S~\x03\xd1`\xa68\x8aE\xdb\x91h.\xc0\xd1\xdc\xdeH\xc7', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fsnap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x10\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x925{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"sequence_number","type":["null","long"],"default":null,"field-id":3},{"name":"file_sequence_number","type":["null","long"],"default":null,"field-id":4},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"content","type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes","field-id":134},{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"doc":"Partition data tuple, schema based on the partition spec","field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"equality_ids","type":["null",{"type":"array","items":"int","element-id":136}],"doc":"Equality comparison field IDs","default":null,"field-id":135},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x022"partition-spec-id\x020\x1ciceberg.schema\xca+{"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":3,"name":"sequence_number","required":false,"type":"long"},{"id":4,"name":"file_sequence_number","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":134,"name":"content","required":true,"type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes"},{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]},"doc":"Partition data tuple, schema based on the partition spec"},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":135,"name":"equality_ids","required":false,"type":{"type":"list","element-id":136,"element":"int","element-required":true},"doc":"Equality comparison field IDs"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x0econtent\x08data\x00h\xff\xed\x8bA \'I\x080\xe5W\xdcr\xf1-\x02\x9c\x035\x8c1N\x031\x10E\xbd\x8e\x0b*\xe0"f\xd7\xbbv\x12\xd3\xa5\xa0\x87\x08\xea\xd1x=C\x90R$\xbb\xde\x0b \xd1\xe6\x08\x1c\x80\x82\x863 -E\xee\xc1)\x10kD~\xf5\xfe\xcc\xd3\x97\xf2\xfb\xe3\xf9\xe7\xf3\xeb\xe5x-\x84x\x97\xe5SK\x81\xbaG\x88\x98\xb0\x8c\xc48lS\x99\xa8Op\xfa\xb4\xdb\xa1O\xd4A\r\x9b\xc8=\x18C\xce\xbb\xc8\xc0\xb5Y\x80E\xaa\x009Dh\x88\x17\x8c\xb6\xf1\x9e\xa7\xa1\xbcV\xe5h\xe3\xf5\x12\xe3\x1c\x97m\xa5}\x13jmi\xba\x06W;m#;\xac\x82EK\x8d\xce\xb2\xb9\xdaa\xb7\x1f(\x9d\xdf\xae\xd6w\x0f7\xf7cq\xb8\x90J\xbe\xcd\xd4\xebLL0\x16j,2\x08%\x84\xccp)\xfe\xa3\xa4\xf9\xeb\xed\xa9+\xef\'G\x9ee\xf1\x17h\xff\xed\x8bA \'I\x080\xe5W\xdcr\xf1-', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2F1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:09 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x16avro.schema\x9e\x1e{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"content","type":"int","doc":"Contents of the manifest: 0=data, 1=deletes","field-id":517},{"name":"sequence_number","type":"long","doc":"Sequence number when the manifest was added","field-id":515},{"name":"min_sequence_number","type":"long","doc":"Lowest sequence number in the manifest","field-id":516},{"name":"added_snapshot_id","type":"long","doc":"Snapshot ID that added the manifest","field-id":503},{"name":"added_data_files_count","type":"int","doc":"Added entry count","field-id":504},{"name":"existing_data_files_count","type":"int","doc":"Existing entry count","field-id":505},{"name":"deleted_data_files_count","type":"int","doc":"Deleted entry count","field-id":506},{"name":"added_rows_count","type":"long","doc":"Added rows count","field-id":512},{"name":"existing_rows_count","type":"long","doc":"Existing rows count","field-id":513},{"name":"deleted_rows_count","type":"long","doc":"Deleted rows count","field-id":514},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507}]}\x14avro.codec\x0edeflate\x16snapshot-id$836654310729966363\x1cformat-version\x022\x1esequence-number\x023\x1ciceberg.schema\xd6\x1f{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":517,"name":"content","required":true,"type":"int","doc":"Contents of the manifest: 0=data, 1=deletes"},{"id":515,"name":"sequence_number","required":true,"type":"long","doc":"Sequence number when the manifest was added"},{"id":516,"name":"min_sequence_number","required":true,"type":"long","doc":"Lowest sequence number in the manifest"},{"id":503,"name":"added_snapshot_id","required":true,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":true,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":true,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":true,"type":"int","doc":"Deleted entry count"},{"id":512,"name":"added_rows_count","required":true,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":true,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":true,"type":"long","doc":"Deleted rows count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"}]}$parent-snapshot-id&4431766687501486370\x00\xbc/\xbf\x81\xc1-\xcf~0d\xa6f\xc9\xc5)\xe5\x06\xf4\x03\xbd\xd0?J\x03A\x14\xc7\xf1q\t\xb9\x82\xa7\x18wfw\xfe\xec\xf32\xc3\x9b7o\x8c\x90 \xecNllD\xb0\xb3\xb4Haie\xa9\xe8\t\x14\x14\xf1\x0cv"\xa27\x90`\x10"^@\xeb\x1f|\xf9\xf0[T\xf5.q\xe4~\'$,X\'\xce8\x9f\x96\xba\xf0P\xc2z\xa1\xe9|(\xdc\x87&LR\x1e\x82\xd6l\xc1\xa6\x1cr\xa3}0\xc8*`\x8e)\xb4\x9c}F\xd3\x02d\xacg\\\xf0\xbbh\x1a\xdbb\xb6Nv\x94\xac4\xc4 \xa3KZ\x82\'`\xf2\x8dC\xad\xe4Lm\xe1~\xbf\xf7>\x11b<\xbe9y\xfe\xb8\xbe\xb8<\xdb\xac\x84\xb8\xdf\x10\xa2\x12\x8b?Wr\xd4\x90Hu\xd2\xb0\xf3\xd2\xa8.Jp\xbc\xf2j\xe5\xc9%\x88\x8d\xf5k\xe5\xdbJ9\x1a\xdd\x9e\x7f.\x1fO_\x0e\x0f\xfeQ\xa9S\xe4\xd6G+\x99@Kc"J0\x8ed\x8e\xd9@cH\x11\xc0\xef/\xab\xea\xf5\xeahy\xf7p\xfc\xb4\xfd\xa3\xfc\x02\xbc/\xbf\x81\xc1-\xcf~0d\xa6f\xc9\xc5)\xe5', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:09 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fsnap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:09 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:09 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:09 GMT, Fri, 04 Apr 2025 18:36:09 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:09 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x0e\x16avro.schema\x9e\x1e{"type":"record","name":"manifest_file","fields":[{"name":"manifest_path","type":"string","doc":"Location URI with FS scheme","field-id":500},{"name":"manifest_length","type":"long","doc":"Total file size in bytes","field-id":501},{"name":"partition_spec_id","type":"int","doc":"Spec ID used to write","field-id":502},{"name":"content","type":"int","doc":"Contents of the manifest: 0=data, 1=deletes","field-id":517},{"name":"sequence_number","type":"long","doc":"Sequence number when the manifest was added","field-id":515},{"name":"min_sequence_number","type":"long","doc":"Lowest sequence number in the manifest","field-id":516},{"name":"added_snapshot_id","type":"long","doc":"Snapshot ID that added the manifest","field-id":503},{"name":"added_data_files_count","type":"int","doc":"Added entry count","field-id":504},{"name":"existing_data_files_count","type":"int","doc":"Existing entry count","field-id":505},{"name":"deleted_data_files_count","type":"int","doc":"Deleted entry count","field-id":506},{"name":"added_rows_count","type":"long","doc":"Added rows count","field-id":512},{"name":"existing_rows_count","type":"long","doc":"Existing rows count","field-id":513},{"name":"deleted_rows_count","type":"long","doc":"Deleted rows count","field-id":514},{"name":"partitions","type":["null",{"type":"array","items":{"type":"record","name":"r508","fields":[{"name":"contains_null","type":"boolean","doc":"True if any file has a null partition value","field-id":509},{"name":"contains_nan","type":["null","boolean"],"doc":"True if any file has a nan partition value","default":null,"field-id":518},{"name":"lower_bound","type":["null","bytes"],"doc":"Partition lower bound for all files","default":null,"field-id":510},{"name":"upper_bound","type":["null","bytes"],"doc":"Partition upper bound for all files","default":null,"field-id":511}]},"element-id":508}],"doc":"Summary for each partition","default":null,"field-id":507}]}\x14avro.codec\x0edeflate\x16snapshot-id&2111356058946522356\x1cformat-version\x022\x1esequence-number\x021\x1ciceberg.schema\xd6\x1f{"type":"struct","schema-id":0,"fields":[{"id":500,"name":"manifest_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":501,"name":"manifest_length","required":true,"type":"long","doc":"Total file size in bytes"},{"id":502,"name":"partition_spec_id","required":true,"type":"int","doc":"Spec ID used to write"},{"id":517,"name":"content","required":true,"type":"int","doc":"Contents of the manifest: 0=data, 1=deletes"},{"id":515,"name":"sequence_number","required":true,"type":"long","doc":"Sequence number when the manifest was added"},{"id":516,"name":"min_sequence_number","required":true,"type":"long","doc":"Lowest sequence number in the manifest"},{"id":503,"name":"added_snapshot_id","required":true,"type":"long","doc":"Snapshot ID that added the manifest"},{"id":504,"name":"added_data_files_count","required":true,"type":"int","doc":"Added entry count"},{"id":505,"name":"existing_data_files_count","required":true,"type":"int","doc":"Existing entry count"},{"id":506,"name":"deleted_data_files_count","required":true,"type":"int","doc":"Deleted entry count"},{"id":512,"name":"added_rows_count","required":true,"type":"long","doc":"Added rows count"},{"id":513,"name":"existing_rows_count","required":true,"type":"long","doc":"Existing rows count"},{"id":514,"name":"deleted_rows_count","required":true,"type":"long","doc":"Deleted rows count"},{"id":507,"name":"partitions","required":false,"type":{"type":"list","element-id":508,"element":{"type":"struct","fields":[{"id":509,"name":"contains_null","required":true,"type":"boolean","doc":"True if any file has a null partition value"},{"id":518,"name":"contains_nan","required":false,"type":"boolean","doc":"True if any file has a nan partition value"},{"id":510,"name":"lower_bound","required":false,"type":"binary","doc":"Partition lower bound for all files"},{"id":511,"name":"upper_bound","required":false,"type":"binary","doc":"Partition upper bound for all files"}]},"element-required":true},"doc":"Summary for each partition"}]}$parent-snapshot-id\x08null\x00\x81\x1a\x91\xc6h\x8exe`\x10\xec\xd8\xe0\xb4:a\x02\xa4\x025\xcc\xd1\t\xc20\x10\x80\xe1\x98}b\x9a\xf6j9\x979.\xc9\x9d\x15*B\x9b:\x80\xe0\x0e\xee\xe1\x0c\x82>\xb8\x87\x0f\xce \x8a\xe0\xeb\xff\xc1\x7f\xb6~\x9b$\xca\xb8\xa1\xcc\x85}\x16\xe5y(\xbe\xc8T\xe8/i\x98\xa7"#\xd5\xd4g\x9d(\x04i\xb1\xcdJZ\x87\x8e\x80\xa5"\xd6\x98\xa9\x11\xed\x94\xa1AT\xf6;)\xfc;\x86\x1c\xa5\xe9b\xeb$ap\x00\x91\x1d\xc2*9\x8d\nXC\xaa\x12\xa2\xdbUK>\x8c\xfbWo\x8c\xb5\xcf\xcb\xf1}\xbd\x9f\x1ekk\xccm\xf1-\xe6\x03\x81\x1a\x91\xc6h\x8exe`\x10\xec\xd8\xe0\xb4:a', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fsnap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:10 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'3', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fversion-hint.text&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:10 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 2,\n "table-uuid" : "13d813c2-9ac5-40bf-9809-0177cfcab88e",\n "location" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa",\n "last-sequence-number" : 1,\n "last-updated-ms" : 1743791768254,\n "last-column-id" : 2,\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 2111356058946522356,\n "refs" : {\n "main" : {\n "snapshot-id" : 2111356058946522356,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "sequence-number" : 1,\n "snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768254,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791768254,\n "snapshot-id" : 2111356058946522356\n } ],\n "metadata-log" : [ ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fv1.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:10 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'{\n "format-version" : 2,\n "table-uuid" : "13d813c2-9ac5-40bf-9809-0177cfcab88e",\n "location" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa",\n "last-sequence-number" : 3,\n "last-updated-ms" : 1743791769334,\n "last-column-id" : 2,\n "current-schema-id" : 0,\n "schemas" : [ {\n "type" : "struct",\n "schema-id" : 0,\n "fields" : [ {\n "id" : 1,\n "name" : "a",\n "required" : false,\n "type" : "long"\n }, {\n "id" : 2,\n "name" : "b",\n "required" : false,\n "type" : "string"\n } ]\n } ],\n "default-spec-id" : 0,\n "partition-specs" : [ {\n "spec-id" : 0,\n "fields" : [ ]\n } ],\n "last-partition-id" : 999,\n "default-sort-order-id" : 0,\n "sort-orders" : [ {\n "order-id" : 0,\n "fields" : [ ]\n } ],\n "properties" : {\n "owner" : "root"\n },\n "current-snapshot-id" : 836654310729966363,\n "refs" : {\n "main" : {\n "snapshot-id" : 836654310729966363,\n "type" : "branch"\n }\n },\n "snapshots" : [ {\n "sequence-number" : 1,\n "snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768254,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "100",\n "total-files-size" : "967",\n "total-data-files" : "1",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro",\n "schema-id" : 0\n }, {\n "sequence-number" : 2,\n "snapshot-id" : 4431766687501486370,\n "parent-snapshot-id" : 2111356058946522356,\n "timestamp-ms" : 1743791768691,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "200",\n "total-files-size" : "1934",\n "total-data-files" : "2",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro",\n "schema-id" : 0\n }, {\n "sequence-number" : 3,\n "snapshot-id" : 836654310729966363,\n "parent-snapshot-id" : 4431766687501486370,\n "timestamp-ms" : 1743791769334,\n "summary" : {\n "operation" : "append",\n "spark.app.id" : "local-1743791753009",\n "added-data-files" : "1",\n "added-records" : "100",\n "added-files-size" : "967",\n "changed-partition-count" : "1",\n "total-records" : "300",\n "total-files-size" : "2901",\n "total-data-files" : "3",\n "total-delete-files" : "0",\n "total-position-deletes" : "0",\n "total-equality-deletes" : "0"\n },\n "manifest-list" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro",\n "schema-id" : 0\n } ],\n "statistics" : [ ],\n "snapshot-log" : [ {\n "timestamp-ms" : 1743791768254,\n "snapshot-id" : 2111356058946522356\n }, {\n "timestamp-ms" : 1743791768691,\n "snapshot-id" : 4431766687501486370\n }, {\n "timestamp-ms" : 1743791769334,\n "snapshot-id" : 836654310729966363\n } ],\n "metadata-log" : [ {\n "timestamp-ms" : 1743791768254,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json"\n }, {\n "timestamp-ms" : 1743791768691,\n "metadata-file" : "/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json"\n } ]\n}', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2Fv3.metadata.json&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:10 [ 633 ] INFO : GETFILESTATUS /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata user.name=root 172.16.1.2:50070 (__init__.py:412, _request) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "GET /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata?user.name=root&op=GETFILESTATUS HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : write_data protocol:http host:hdfs1 port:50070 path: /iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro user:root, principal:None (hdfs_api.py:256, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50070/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro?op=CREATE', 'allow_redirects': False, 'headers': {'host': '172.16.1.2'}, 'params': {'overwrite': 'true'}, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50070 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50070 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro?op=CREATE&overwrite=true HTTP/1.1" 307 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : HDFS api response:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'http://hdfs1:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:284, write_file) 2025-04-04 18:36:10 [ 633 ] DEBUG : CALL: {'url': 'http://172.16.1.2:50075/webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true', 'data': b'Obj\x01\x10\x0cschema\xa4\x02{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"a","required":false,"type":"long"},{"id":2,"name":"b","required":false,"type":"string"}]}\x16avro.schema\x925{"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"sequence_number","type":["null","long"],"default":null,"field-id":3},{"name":"file_sequence_number","type":["null","long"],"default":null,"field-id":4},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"content","type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes","field-id":134},{"name":"file_path","type":"string","doc":"Location URI with FS scheme","field-id":100},{"name":"file_format","type":"string","doc":"File format name: avro, orc, or parquet","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"doc":"Partition data tuple, schema based on the partition spec","field-id":102},{"name":"record_count","type":"long","doc":"Number of records in the file","field-id":103},{"name":"file_size_in_bytes","type":"long","doc":"Total file size in bytes","field-id":104},{"name":"column_sizes","type":["null",{"type":"array","items":{"type":"record","name":"k117_v118","fields":[{"name":"key","type":"int","field-id":117},{"name":"value","type":"long","field-id":118}]},"logicalType":"map"}],"doc":"Map of column id to total size on disk","default":null,"field-id":108},{"name":"value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k119_v120","fields":[{"name":"key","type":"int","field-id":119},{"name":"value","type":"long","field-id":120}]},"logicalType":"map"}],"doc":"Map of column id to total count, including null and NaN","default":null,"field-id":109},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"doc":"Map of column id to null value count","default":null,"field-id":110},{"name":"nan_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k138_v139","fields":[{"name":"key","type":"int","field-id":138},{"name":"value","type":"long","field-id":139}]},"logicalType":"map"}],"doc":"Map of column id to number of NaN values in the column","default":null,"field-id":137},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"doc":"Map of column id to lower bound","default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"doc":"Map of column id to upper bound","default":null,"field-id":128},{"name":"key_metadata","type":["null","bytes"],"doc":"Encryption key metadata blob","default":null,"field-id":131},{"name":"split_offsets","type":["null",{"type":"array","items":"long","element-id":133}],"doc":"Splittable offsets","default":null,"field-id":132},{"name":"equality_ids","type":["null",{"type":"array","items":"int","element-id":136}],"doc":"Equality comparison field IDs","default":null,"field-id":135},{"name":"sort_order_id","type":["null","int"],"doc":"Sort order ID","default":null,"field-id":140}]},"field-id":2}]}\x14avro.codec\x0edeflate\x1cformat-version\x022"partition-spec-id\x020\x1ciceberg.schema\xca+{"type":"struct","schema-id":0,"fields":[{"id":0,"name":"status","required":true,"type":"int"},{"id":1,"name":"snapshot_id","required":false,"type":"long"},{"id":3,"name":"sequence_number","required":false,"type":"long"},{"id":4,"name":"file_sequence_number","required":false,"type":"long"},{"id":2,"name":"data_file","required":true,"type":{"type":"struct","fields":[{"id":134,"name":"content","required":true,"type":"int","doc":"Contents of the file: 0=data, 1=position deletes, 2=equality deletes"},{"id":100,"name":"file_path","required":true,"type":"string","doc":"Location URI with FS scheme"},{"id":101,"name":"file_format","required":true,"type":"string","doc":"File format name: avro, orc, or parquet"},{"id":102,"name":"partition","required":true,"type":{"type":"struct","fields":[]},"doc":"Partition data tuple, schema based on the partition spec"},{"id":103,"name":"record_count","required":true,"type":"long","doc":"Number of records in the file"},{"id":104,"name":"file_size_in_bytes","required":true,"type":"long","doc":"Total file size in bytes"},{"id":108,"name":"column_sizes","required":false,"type":{"type":"map","key-id":117,"key":"int","value-id":118,"value":"long","value-required":true},"doc":"Map of column id to total size on disk"},{"id":109,"name":"value_counts","required":false,"type":{"type":"map","key-id":119,"key":"int","value-id":120,"value":"long","value-required":true},"doc":"Map of column id to total count, including null and NaN"},{"id":110,"name":"null_value_counts","required":false,"type":{"type":"map","key-id":121,"key":"int","value-id":122,"value":"long","value-required":true},"doc":"Map of column id to null value count"},{"id":137,"name":"nan_value_counts","required":false,"type":{"type":"map","key-id":138,"key":"int","value-id":139,"value":"long","value-required":true},"doc":"Map of column id to number of NaN values in the column"},{"id":125,"name":"lower_bounds","required":false,"type":{"type":"map","key-id":126,"key":"int","value-id":127,"value":"binary","value-required":true},"doc":"Map of column id to lower bound"},{"id":128,"name":"upper_bounds","required":false,"type":{"type":"map","key-id":129,"key":"int","value-id":130,"value":"binary","value-required":true},"doc":"Map of column id to upper bound"},{"id":131,"name":"key_metadata","required":false,"type":"binary","doc":"Encryption key metadata blob"},{"id":132,"name":"split_offsets","required":false,"type":{"type":"list","element-id":133,"element":"long","element-required":true},"doc":"Splittable offsets"},{"id":135,"name":"equality_ids","required":false,"type":{"type":"list","element-id":136,"element":"int","element-required":true},"doc":"Equality comparison field IDs"},{"id":140,"name":"sort_order_id","required":false,"type":"int","doc":"Sort order ID"}]}}]}\x1cpartition-spec\x04[]\x0econtent\x08data\x00C\xde\xee\xf2\xa7#\x9d\xdc2q;\x1a\xf3[\xc4M\x02\x9c\x035\x8c1N\xc30\x18\x85\x1d\xd7\x03\x13t\xe3\x14&ql\xc7\xf1\xc8\xc0\x0e\x08\xe6_\x7f\xe3\xdf\xb4R\x87\x928\xb7\xa8z\x02\xc4\x01:0 \xc1\x19r\x0bN\xc0\x1d\x88Q\xfb\xa6\xef\xe9}z\x9c\x7f\xef\x7f~\xbf\x8e\x9fo\xd7\x8c\xb1\x0f^n:ZQ\xff\x02\x01\x13\x96\x81"\x8e\xdbT&\x1a\x12\x9c\x97n;\x0e\x89z\xa8a\x1d\xe2\x00J\x91\xf56D\x88\xb5r`\x90*\xc0\xb8\n\xa0)\xba\x88F{\x1f\xe7\xa3\xfcV\xe5\xc8Z\xcb\xc6\xd9\xe8t[K\xd7TZ\x9aV9\x89\xd6\x18\xd9\xd8FkjQ\xb5\xe8d\x96\xd5\xcd\x0e\xfb\xd7\x91\xd2\xe5\xfd\xed\xe3\xc3\xf3\xdd\xd3T\x1c\xae\xb8\xe0\xc7\x85x_\xb0\x19\xa6BLE\x06&\x18\xe3\x19\x96\xec\x14\xc1\xd5\x7f\xef\xce]x?;\xfc"\x8b\x7fC\xde\xee\xf2\xa7#\x9d\xdc2q;\x1a\xf3[\xc4M', 'headers': {'content-type': 'text/plain', 'host': '172.16.1.2'}, 'params': {'file': '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro', 'user.name': 'root'}, 'allow_redirects': False, 'verify': False, 'auth': None} (hdfs_api.py:131, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : Starting new HTTP connection (1): 172.16.1.2:50075 (connectionpool.py:245, _new_conn) 2025-04-04 18:36:10 [ 633 ] DEBUG : http://172.16.1.2:50075 "PUT /webhdfs/v1/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro?op=CREATE&namenoderpcaddress=hdfs1:9000&overwrite=true&file=%2Ficeberg_data%2Fdefault%2Ftest_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa%2Fmetadata%2F4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro&user.name=root HTTP/1.1" 201 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:10 [ 633 ] DEBUG : response_data:b'' headers:{'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:133, req_wrapper) 2025-04-04 18:36:10 [ 633 ] DEBUG : b'' {'Cache-Control': 'no-cache', 'Expires': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Date': 'Fri, 04 Apr 2025 18:36:10 GMT, Fri, 04 Apr 2025 18:36:10 GMT', 'Pragma': 'no-cache, no-cache', 'Content-Type': 'application/octet-stream', 'Location': 'webhdfs://hdfs1:9000/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro', 'Content-Length': '0', 'Server': 'Jetty(6.1.26)'} (hdfs_api.py:309, write_file) 2025-04-04 18:36:10 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro'] (test.py:645, add_df) 2025-04-04 18:36:10 [ 633 ] INFO : Setup complete. files: ['/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-23-675f7382-7603-4817-a544-65633e8a18a7-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-19-8ad6a8c0-93b2-4e00-b525-4df5a0b4a4e3-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/data/00000-21-dfa16d76-562c-4c8e-a599-7a4e7451f900-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/eb19dc08-4e67-408b-96e5-4107c6d9b257-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-4431766687501486370-1-eb19dc08-4e67-408b-96e5-4107c6d9b257.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/1dbe37b5-ec91-44ba-946c-fbf4924c0c99-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-836654310729966363-1-4253af56-8cd5-4ce9-b6d1-97c9ec726a10.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/snap-2111356058946522356-1-1dbe37b5-ec91-44ba-946c-fbf4924c0c99.avro', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/v3.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/metadata/4253af56-8cd5-4ce9-b6d1-97c9ec726a10-m0.avro'] (test.py:653, test_cluster_table_function) 2025-04-04 18:36:10 [ 633 ] DEBUG : Executing query SELECT * FROM system.clusters on node1 (cluster.py:3677, query) 2025-04-04 18:36:10 [ 633 ] INFO : Clusters setup: cluster_simple 1 1 0 1 node1 172.16.1.9 9000 1 default 0 0 0 \N \N \N cluster_simple 1 1 0 2 node2 172.16.1.8 9000 0 default 0 0 0 \N \N \N cluster_simple 1 1 0 3 node3 172.16.1.10 9000 0 default 0 0 0 \N \N \N (test.py:657, test_cluster_table_function) 2025-04-04 18:36:10 [ 633 ] DEBUG : Executing query SELECT * FROM icebergHDFS(hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/', format=Parquet, url = 'hdfs://hdfs1:9000/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:10 [ 633 ] DEBUG : Executing query SELECT * FROM icebergHDFSCluster('cluster_simple', hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/', format=Parquet, url = 'hdfs://hdfs1:9000/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:10 [ 633 ] DEBUG : Executing query SELECT * FROM icebergHDFS(hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/', format=Parquet, url = 'hdfs://hdfs1:9000/') SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:10 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa; CREATE TABLE test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa ENGINE=IcebergHDFS(hdfs, filename = 'iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/', format=Parquet, url = 'hdfs://hdfs1:9000/') SETTINGS object_storage_cluster = 'cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:10 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa on node1 (cluster.py:3677, query) 2025-04-04 18:36:11 [ 633 ] DEBUG : Executing query SELECT * FROM remote('node2', icebergHDFSCluster('cluster_simple', hdfs, filename= 'iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/', format=Parquet, url = 'hdfs://hdfs1:9000/') ) on node1 (cluster.py:3677, query) 2025-04-04 18:36:11 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS `test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa` SYNC on node1 (cluster.py:3677, query) 2025-04-04 18:36:11 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa; CREATE TABLE test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa ENGINE=IcebergHDFS(hdfs, filename = 'iceberg_data/default/test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa/', format=Parquet, url = 'hdfs://hdfs1:9000/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:11 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa on node1 (cluster.py:3677, query) 2025-04-04 18:36:11 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_hdfs_11e595df_f217_4ae0_afbd_3ef7fa4399fa SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) ______________________ test_cluster_table_function[s3-1] _______________________ started_cluster = format_version = '1', storage_type = 's3' @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs"]) def test_cluster_table_function(started_cluster, format_version, storage_type): if is_arm() and storage_type == "hdfs": pytest.skip("Disabled test IcebergHDFS for aarch64") instance = started_cluster.instances["node1"] spark = started_cluster.spark_session TABLE_NAME = ( "test_iceberg_cluster_" + format_version + "_" + storage_type + "_" + get_uuid_str() ) def add_df(mode): write_iceberg_from_df( spark, generate_data(spark, 0, 100), TABLE_NAME, mode=mode, format_version=format_version, ) files = default_upload_directory( started_cluster, storage_type, f"/iceberg_data/default/{TABLE_NAME}/", f"/iceberg_data/default/{TABLE_NAME}/", ) logging.info(f"Adding another dataframe. result files: {files}") return files files = add_df(mode="overwrite") for i in range(1, len(started_cluster.instances)): files = add_df(mode="append") logging.info(f"Setup complete. files: {files}") assert len(files) == 5 + 4 * (len(started_cluster.instances) - 1) clusters = instance.query(f"SELECT * FROM system.clusters") logging.info(f"Clusters setup: {clusters}") # Regular Query only node1 table_function_expr = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True ) select_regular = ( instance.query(f"SELECT * FROM {table_function_expr}").strip().split() ) # Cluster Query with node1 as coordinator table_function_expr_cluster = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True, run_on_cluster=True, ) query_id_cluster = str(uuid.uuid4()) select_cluster = ( instance.query( f"SELECT * FROM {table_function_expr_cluster}", query_id=query_id_cluster ) .strip() .split() ) # Cluster Query with node1 as coordinator with alternative syntax query_id_cluster_alt_syntax = str(uuid.uuid4()) select_cluster_alt_syntax = ( instance.query( f""" SELECT * FROM {table_function_expr} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_cluster_alt_syntax, ) .strip() .split() ) create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster, object_storage_cluster='cluster_simple') query_id_cluster_table_engine = str(uuid.uuid4()) select_cluster_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_cluster_table_engine, ) .strip() .split() ) select_remote_cluster = ( instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") .strip() .split() ) instance.query(f"DROP TABLE IF EXISTS `{TABLE_NAME}` SYNC") create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster) query_id_pure_table_engine = str(uuid.uuid4()) select_pure_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_pure_table_engine, ) .strip() .split() ) query_id_pure_table_engine_cluster = str(uuid.uuid4()) select_pure_table_engine_cluster = ( instance.query( f""" SELECT * FROM {TABLE_NAME} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_pure_table_engine_cluster, ) .strip() .split() ) # Simple size check assert len(select_regular) == 600 assert len(select_cluster) == 600 assert len(select_cluster_alt_syntax) == 600 > assert len(select_cluster_table_engine) == 600 E AssertionError: assert 1800 == 600 E + where 1800 = len(['0', '1', '1', '2', '2', '3', ...]) test_storage_iceberg/test.py:747: AssertionError ----------------------------- Captured stdout call ----------------------------- 25/04/04 18:36:11 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:11 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:11 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:11 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} Upload to bucket: None {} Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} Upload to bucket: None Upload to bucket: None {} Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:12 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} Upload to bucket: None Upload to bucket: None Upload to bucket: None {} Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None ------------------------------ Captured log call ------------------------------- 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro526 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o526 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro527 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ylo528 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o528 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro528 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro529 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o527 toDF ro529 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro530 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro531 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o531 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro532 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ylo533 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o533 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro533 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro534 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o532 toDF ro534 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro535 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o535 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro536 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro537 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o537 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro538 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o538 get e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro539 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro540 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yao541 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o540 applyModifiableSettings ro539 ro541 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro542 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o536 cast ro542 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro543 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o535 withColumn sb ro543 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro544 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro545 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro546 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ylo547 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o547 add ro546 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro547 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro548 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro548 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro549 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o545 over ro549 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro550 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o530 withColumn srow_index ro550 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro551 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro552 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro553 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ylo554 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o554 add ro553 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro554 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro555 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro555 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro556 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o552 over ro556 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro557 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o544 withColumn srow_index ro557 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro558 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ylo559 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o559 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro559 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro560 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o551 join ro558 ro560 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro561 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o561 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro562 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o562 writeTo stest_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro563 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o563 tableProperty sformat-version s1 e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro564 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o563 using siceberg e (clientserver.py:501, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Answer received: !yro565 (clientserver.py:512, send_command) 2025-04-04 18:36:11 [ 633 ] DEBUG : Command to send: c o563 create e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-25-88f3218b-a9b4-4a7f-87bd-c56ad0874ca6-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/version-hint.text HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-4944424703226519386-1-6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v1.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-25-88f3218b-a9b4-4a7f-87bd-c56ad0874ca6-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-4944424703226519386-1-6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro566 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o566 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro567 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo568 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o568 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro568 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro569 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o567 toDF ro569 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro570 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro571 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o571 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro572 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo573 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o573 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro573 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro574 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o572 toDF ro574 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro575 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o575 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro576 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro577 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o577 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro578 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o578 get e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro579 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro580 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yao581 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o580 applyModifiableSettings ro579 ro581 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro582 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o576 cast ro582 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro583 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o575 withColumn sb ro583 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro584 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro585 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro586 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo587 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o587 add ro586 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro587 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro588 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro588 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro589 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o585 over ro589 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro590 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o570 withColumn srow_index ro590 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro591 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro592 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro593 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo594 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o594 add ro593 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro594 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro595 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro595 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro596 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o592 over ro596 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro597 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o584 withColumn srow_index ro597 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro598 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo599 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o599 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro599 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro600 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o591 join ro598 ro600 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro601 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o601 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro602 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o602 writeTo stest_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro603 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o603 append e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o446 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o423 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o454 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o468 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o470 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o469 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o463 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o528 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o533 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o541 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o547 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o554 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o559 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o17 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o16 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o14 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o25 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o153 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o269 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o385 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o501 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o526 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o527 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o529 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o530 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o531 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o532 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o534 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o535 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o536 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o537 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o538 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o540 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o542 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o543 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o544 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o545 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o546 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o548 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o549 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o550 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o551 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o552 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o553 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o555 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o556 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o557 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o558 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o560 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o561 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o564 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o565 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o568 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o573 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o581 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o587 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o563 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o566 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o567 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o569 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o570 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o571 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o572 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o574 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o575 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o576 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o577 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o578 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o580 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o582 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o583 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o585 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o586 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o588 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o589 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o590 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o594 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: m d o599 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-27-1d1d643d-1252-4239-9f90-d444b91d2202-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-25-88f3218b-a9b4-4a7f-87bd-c56ad0874ca6-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v2.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/debcf952-9e99-41c4-b5c2-126026245f7c-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-395693523090131734-1-debcf952-9e99-41c4-b5c2-126026245f7c.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/version-hint.text HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-4944424703226519386-1-6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v1.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-27-1d1d643d-1252-4239-9f90-d444b91d2202-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-25-88f3218b-a9b4-4a7f-87bd-c56ad0874ca6-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/debcf952-9e99-41c4-b5c2-126026245f7c-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-395693523090131734-1-debcf952-9e99-41c4-b5c2-126026245f7c.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-4944424703226519386-1-6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro604 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o604 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro605 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo606 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o606 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro606 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro607 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o605 toDF ro607 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro608 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro609 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o609 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro610 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo611 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o611 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro611 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro612 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o610 toDF ro612 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro613 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o613 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro614 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro615 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o615 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro616 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o616 get e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro617 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro618 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yao619 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o618 applyModifiableSettings ro617 ro619 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro620 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o614 cast ro620 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro621 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o613 withColumn sb ro621 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro622 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro623 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro624 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo625 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o625 add ro624 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro625 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro626 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro626 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro627 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o623 over ro627 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro628 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o608 withColumn srow_index ro628 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro629 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro630 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro631 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo632 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o632 add ro631 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro632 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro633 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro633 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro634 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o630 over ro634 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro635 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o622 withColumn srow_index ro635 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro636 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ylo637 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o637 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro637 e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro638 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o629 join ro636 ro638 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro639 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o639 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro640 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o640 writeTo stest_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yro641 (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Command to send: c o641 append e (clientserver.py:501, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-27-1d1d643d-1252-4239-9f90-d444b91d2202-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-29-d9cda629-c9e9-4014-be1f-cf46c1c70333-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-25-88f3218b-a9b4-4a7f-87bd-c56ad0874ca6-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v2.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-5215258184428002463-1-ad6c97e7-73ca-4bac-8660-08d7e8ffd300.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/debcf952-9e99-41c4-b5c2-126026245f7c-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-395693523090131734-1-debcf952-9e99-41c4-b5c2-126026245f7c.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/version-hint.text HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-4944424703226519386-1-6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/ad6c97e7-73ca-4bac-8660-08d7e8ffd300-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v1.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v3.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:12 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-27-1d1d643d-1252-4239-9f90-d444b91d2202-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-29-d9cda629-c9e9-4014-be1f-cf46c1c70333-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-25-88f3218b-a9b4-4a7f-87bd-c56ad0874ca6-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-5215258184428002463-1-ad6c97e7-73ca-4bac-8660-08d7e8ffd300.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/debcf952-9e99-41c4-b5c2-126026245f7c-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-395693523090131734-1-debcf952-9e99-41c4-b5c2-126026245f7c.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-4944424703226519386-1-6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/ad6c97e7-73ca-4bac-8660-08d7e8ffd300-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v3.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:12 [ 633 ] INFO : Setup complete. files: ['/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-27-1d1d643d-1252-4239-9f90-d444b91d2202-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-29-d9cda629-c9e9-4014-be1f-cf46c1c70333-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/data/00000-25-88f3218b-a9b4-4a7f-87bd-c56ad0874ca6-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-5215258184428002463-1-ad6c97e7-73ca-4bac-8660-08d7e8ffd300.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/debcf952-9e99-41c4-b5c2-126026245f7c-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-395693523090131734-1-debcf952-9e99-41c4-b5c2-126026245f7c.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/snap-4944424703226519386-1-6ea5f0c2-57e7-4b1e-8d84-99bf7122ef3d.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/ad6c97e7-73ca-4bac-8660-08d7e8ffd300-m0.avro', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/metadata/v3.metadata.json'] (test.py:653, test_cluster_table_function) 2025-04-04 18:36:12 [ 633 ] DEBUG : Executing query SELECT * FROM system.clusters on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] INFO : Clusters setup: cluster_simple 1 1 0 1 node1 172.16.1.9 9000 1 default 0 0 0 \N \N \N cluster_simple 1 1 0 2 node2 172.16.1.8 9000 0 default 0 0 0 \N \N \N cluster_simple 1 1 0 3 node3 172.16.1.10 9000 0 default 0 0 0 \N \N \N (test.py:657, test_cluster_table_function) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query SELECT * FROM icebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/', format=Parquet, url = 'http://minio1:9001/root/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query SELECT * FROM icebergS3Cluster('cluster_simple', s3, filename = 'iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/', format=Parquet, url = 'http://minio1:9001/root/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query SELECT * FROM icebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/', format=Parquet, url = 'http://minio1:9001/root/') SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd; CREATE TABLE test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd ENGINE=IcebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/', format=Parquet, url = 'http://minio1:9001/root/') SETTINGS object_storage_cluster = 'cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o606 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o611 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o619 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o595 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o596 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o597 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o598 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o600 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o601 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o602 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o603 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o604 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o605 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o607 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o609 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o610 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o612 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o615 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o616 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o618 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o625 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o632 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Command to send: m d o637 e (clientserver.py:501, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query SELECT * FROM remote('node2',icebergS3Cluster('cluster_simple', s3, filename = 'iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/', format=Parquet, url = 'http://minio1:9001/root/')) on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS `test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd` SYNC on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd; CREATE TABLE test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd ENGINE=IcebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd/', format=Parquet, url = 'http://minio1:9001/root/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:13 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd on node1 (cluster.py:3677, query) 2025-04-04 18:36:14 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_1_s3_e2174a0b_44ff_4718_8e68_3fbc54e45edd SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) ______________________ test_cluster_table_function[s3-2] _______________________ started_cluster = format_version = '2', storage_type = 's3' @pytest.mark.parametrize("format_version", ["1", "2"]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs"]) def test_cluster_table_function(started_cluster, format_version, storage_type): if is_arm() and storage_type == "hdfs": pytest.skip("Disabled test IcebergHDFS for aarch64") instance = started_cluster.instances["node1"] spark = started_cluster.spark_session TABLE_NAME = ( "test_iceberg_cluster_" + format_version + "_" + storage_type + "_" + get_uuid_str() ) def add_df(mode): write_iceberg_from_df( spark, generate_data(spark, 0, 100), TABLE_NAME, mode=mode, format_version=format_version, ) files = default_upload_directory( started_cluster, storage_type, f"/iceberg_data/default/{TABLE_NAME}/", f"/iceberg_data/default/{TABLE_NAME}/", ) logging.info(f"Adding another dataframe. result files: {files}") return files files = add_df(mode="overwrite") for i in range(1, len(started_cluster.instances)): files = add_df(mode="append") logging.info(f"Setup complete. files: {files}") assert len(files) == 5 + 4 * (len(started_cluster.instances) - 1) clusters = instance.query(f"SELECT * FROM system.clusters") logging.info(f"Clusters setup: {clusters}") # Regular Query only node1 table_function_expr = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True ) select_regular = ( instance.query(f"SELECT * FROM {table_function_expr}").strip().split() ) # Cluster Query with node1 as coordinator table_function_expr_cluster = get_creation_expression( storage_type, TABLE_NAME, started_cluster, table_function=True, run_on_cluster=True, ) query_id_cluster = str(uuid.uuid4()) select_cluster = ( instance.query( f"SELECT * FROM {table_function_expr_cluster}", query_id=query_id_cluster ) .strip() .split() ) # Cluster Query with node1 as coordinator with alternative syntax query_id_cluster_alt_syntax = str(uuid.uuid4()) select_cluster_alt_syntax = ( instance.query( f""" SELECT * FROM {table_function_expr} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_cluster_alt_syntax, ) .strip() .split() ) create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster, object_storage_cluster='cluster_simple') query_id_cluster_table_engine = str(uuid.uuid4()) select_cluster_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_cluster_table_engine, ) .strip() .split() ) select_remote_cluster = ( instance.query(f"SELECT * FROM remote('node2',{table_function_expr_cluster})") .strip() .split() ) instance.query(f"DROP TABLE IF EXISTS `{TABLE_NAME}` SYNC") create_iceberg_table(storage_type, instance, TABLE_NAME, started_cluster) query_id_pure_table_engine = str(uuid.uuid4()) select_pure_table_engine = ( instance.query( f""" SELECT * FROM {TABLE_NAME} """, query_id=query_id_pure_table_engine, ) .strip() .split() ) query_id_pure_table_engine_cluster = str(uuid.uuid4()) select_pure_table_engine_cluster = ( instance.query( f""" SELECT * FROM {TABLE_NAME} SETTINGS object_storage_cluster='cluster_simple' """, query_id=query_id_pure_table_engine_cluster, ) .strip() .split() ) # Simple size check assert len(select_regular) == 600 assert len(select_cluster) == 600 assert len(select_cluster_alt_syntax) == 600 > assert len(select_cluster_table_engine) == 600 E AssertionError: assert 1800 == 600 E + where 1800 = len(['0', '1', '1', '2', '2', '3', ...]) test_storage_iceberg/test.py:747: AssertionError ----------------------------- Captured stdout call ----------------------------- 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} Upload to bucket: None {} Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} Upload to bucket: None Upload to bucket: None {} Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 25/04/04 18:36:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. {} {} {} Upload to bucket: None Upload to bucket: None Upload to bucket: None {} Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None Upload to bucket: None ------------------------------ Captured log call ------------------------------- 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro642 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o642 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro643 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo644 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o644 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro644 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro645 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o643 toDF ro645 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro646 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro647 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o647 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro648 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo649 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o649 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro649 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro650 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o648 toDF ro650 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro651 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o651 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro652 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro653 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o653 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro654 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o654 get e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro655 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro656 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yao657 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o656 applyModifiableSettings ro655 ro657 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro658 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o652 cast ro658 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro659 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o651 withColumn sb ro659 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro660 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro661 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro662 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo663 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o663 add ro662 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro663 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro664 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro664 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro665 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o661 over ro665 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro666 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o646 withColumn srow_index ro666 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro667 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro668 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro669 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo670 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o670 add ro669 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro670 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro671 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro671 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro672 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o668 over ro672 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro673 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o660 withColumn srow_index ro673 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro674 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo675 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o675 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro675 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro676 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o667 join ro674 ro676 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro677 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o677 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro678 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o678 writeTo stest_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro679 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o679 tableProperty sformat-version s2 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro680 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o679 using siceberg e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro681 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o679 create e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-31-81a55ca0-cbc8-40f5-a033-acf6141d2580-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/9024a302-f9aa-48ce-8ef8-67003fbf7323-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-3421437268171227855-1-9024a302-f9aa-48ce-8ef8-67003fbf7323.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/version-hint.text HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v1.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-31-81a55ca0-cbc8-40f5-a033-acf6141d2580-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/9024a302-f9aa-48ce-8ef8-67003fbf7323-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-3421437268171227855-1-9024a302-f9aa-48ce-8ef8-67003fbf7323.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro682 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o682 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro683 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo684 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o684 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro684 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro685 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o683 toDF ro685 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro686 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro687 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o687 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro688 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo689 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o689 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro689 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro690 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o688 toDF ro690 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro691 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o691 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro692 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro693 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o693 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro694 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o694 get e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro695 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro696 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yao697 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o696 applyModifiableSettings ro695 ro697 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro698 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o692 cast ro698 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro699 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o691 withColumn sb ro699 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro700 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro701 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o621 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro702 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o622 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o623 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o624 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o626 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o627 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o628 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o629 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o630 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o631 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o633 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o634 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o635 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o636 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o638 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo703 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o703 add ro702 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o639 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro703 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro704 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o640 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro704 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o641 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro705 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o701 over ro705 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o584 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro706 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o686 withColumn srow_index ro706 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o591 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o593 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o592 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o579 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o620 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o608 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o613 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o614 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o644 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o649 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o657 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro707 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o663 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o670 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o675 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o642 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o643 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro708 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o645 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o646 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o647 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o648 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o650 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro709 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o651 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o652 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o653 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o654 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o656 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o658 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o659 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o660 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o661 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o662 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o664 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o665 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o666 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o667 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo710 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o710 add ro709 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o668 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro710 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o669 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro711 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro711 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o671 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro712 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o708 over ro712 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o672 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro713 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o700 withColumn srow_index ro713 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o673 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o674 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o676 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o677 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o680 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o681 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o684 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o689 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o697 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o703 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o679 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o682 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o683 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro714 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o685 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o686 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o687 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o688 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo715 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o715 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o690 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro715 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o691 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro716 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o707 join ro714 ro716 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o692 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o693 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o694 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o696 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o698 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o699 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o701 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o702 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o704 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o705 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o706 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o710 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: m d o715 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro717 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o717 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro718 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o718 writeTo stest_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro719 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o719 append e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-33-9f87aba4-40db-467f-b27a-aa209b465dd8-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-31-81a55ca0-cbc8-40f5-a033-acf6141d2580-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v2.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/9024a302-f9aa-48ce-8ef8-67003fbf7323-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-3421437268171227855-1-9024a302-f9aa-48ce-8ef8-67003fbf7323.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/0e1551c5-9c2d-492a-af33-e5cc6d383373-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-7402174360314822335-1-0e1551c5-9c2d-492a-af33-e5cc6d383373.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/version-hint.text HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v1.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:14 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-33-9f87aba4-40db-467f-b27a-aa209b465dd8-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-31-81a55ca0-cbc8-40f5-a033-acf6141d2580-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/9024a302-f9aa-48ce-8ef8-67003fbf7323-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-3421437268171227855-1-9024a302-f9aa-48ce-8ef8-67003fbf7323.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/0e1551c5-9c2d-492a-af33-e5cc6d383373-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-7402174360314822335-1-0e1551c5-9c2d-492a-af33-e5cc6d383373.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v1.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro720 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o720 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro721 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo722 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o722 add sa e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro722 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro723 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o721 toDF ro723 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro724 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o50 sc e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro725 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o725 defaultParallelism e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro726 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo727 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o727 add sb e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro727 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro728 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o726 toDF ro728 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro729 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o729 apply sb e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro730 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro731 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o731 isDefined e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro732 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o732 get e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro733 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro734 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yao735 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o734 applyModifiableSettings ro733 ro735 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o61 parseDataType s"string" e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro736 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o730 cast ro736 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro737 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o729 withColumn sb ro737 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro738 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro739 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro740 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo741 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o741 add ro740 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro741 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro742 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro742 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro743 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o739 over ro743 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro744 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o724 withColumn srow_index ro744 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro745 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro746 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro747 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo748 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o748 add ro747 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro748 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro749 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro749 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro750 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o746 over ro750 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro751 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o738 withColumn srow_index ro751 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro752 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ylo753 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o753 add srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro753 e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro754 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o745 join ro752 ro754 sinner e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro755 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o755 drop srow_index e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro756 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o756 writeTo stest_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b e (clientserver.py:501, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Answer received: !yro757 (clientserver.py:512, send_command) 2025-04-04 18:36:14 [ 633 ] DEBUG : Command to send: c o757 append e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-33-9f87aba4-40db-467f-b27a-aa209b465dd8-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-31-81a55ca0-cbc8-40f5-a033-acf6141d2580-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-35-fc3b1d69-b3bb-4c29-9974-16dd6d87fe0e-00001.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v2.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/9024a302-f9aa-48ce-8ef8-67003fbf7323-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-3421437268171227855-1-9024a302-f9aa-48ce-8ef8-67003fbf7323.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-2479157474636360612-1-b80cfee5-bdd1-4cd1-ba7c-008a9cc0b80b.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/0e1551c5-9c2d-492a-af33-e5cc6d383373-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-7402174360314822335-1-0e1551c5-9c2d-492a-af33-e5cc6d383373.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/version-hint.text HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v1.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/b80cfee5-bdd1-4cd1-ba7c-008a9cc0b80b-m0.avro HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] DEBUG : http://172.16.1.5:9001 "PUT /root//iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v3.metadata.json HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-04 18:36:15 [ 633 ] INFO : Adding another dataframe. result files: ['/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-33-9f87aba4-40db-467f-b27a-aa209b465dd8-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-31-81a55ca0-cbc8-40f5-a033-acf6141d2580-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-35-fc3b1d69-b3bb-4c29-9974-16dd6d87fe0e-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/9024a302-f9aa-48ce-8ef8-67003fbf7323-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-3421437268171227855-1-9024a302-f9aa-48ce-8ef8-67003fbf7323.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-2479157474636360612-1-b80cfee5-bdd1-4cd1-ba7c-008a9cc0b80b.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/0e1551c5-9c2d-492a-af33-e5cc6d383373-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-7402174360314822335-1-0e1551c5-9c2d-492a-af33-e5cc6d383373.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/b80cfee5-bdd1-4cd1-ba7c-008a9cc0b80b-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v3.metadata.json'] (test.py:645, add_df) 2025-04-04 18:36:15 [ 633 ] INFO : Setup complete. files: ['/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-33-9f87aba4-40db-467f-b27a-aa209b465dd8-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-31-81a55ca0-cbc8-40f5-a033-acf6141d2580-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/data/00000-35-fc3b1d69-b3bb-4c29-9974-16dd6d87fe0e-00001.parquet', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v2.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/9024a302-f9aa-48ce-8ef8-67003fbf7323-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-3421437268171227855-1-9024a302-f9aa-48ce-8ef8-67003fbf7323.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-2479157474636360612-1-b80cfee5-bdd1-4cd1-ba7c-008a9cc0b80b.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/0e1551c5-9c2d-492a-af33-e5cc6d383373-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/snap-7402174360314822335-1-0e1551c5-9c2d-492a-af33-e5cc6d383373.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/version-hint.text', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v1.metadata.json', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/b80cfee5-bdd1-4cd1-ba7c-008a9cc0b80b-m0.avro', '/iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/metadata/v3.metadata.json'] (test.py:653, test_cluster_table_function) 2025-04-04 18:36:15 [ 633 ] DEBUG : Executing query SELECT * FROM system.clusters on node1 (cluster.py:3677, query) 2025-04-04 18:36:15 [ 633 ] INFO : Clusters setup: cluster_simple 1 1 0 1 node1 172.16.1.9 9000 1 default 0 0 0 \N \N \N cluster_simple 1 1 0 2 node2 172.16.1.8 9000 0 default 0 0 0 \N \N \N cluster_simple 1 1 0 3 node3 172.16.1.10 9000 0 default 0 0 0 \N \N \N (test.py:657, test_cluster_table_function) 2025-04-04 18:36:15 [ 633 ] DEBUG : Executing query SELECT * FROM icebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/', format=Parquet, url = 'http://minio1:9001/root/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:15 [ 633 ] DEBUG : Executing query SELECT * FROM icebergS3Cluster('cluster_simple', s3, filename = 'iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/', format=Parquet, url = 'http://minio1:9001/root/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:15 [ 633 ] DEBUG : Executing query SELECT * FROM icebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/', format=Parquet, url = 'http://minio1:9001/root/') SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o722 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o727 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o735 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o741 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o711 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o712 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o713 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o714 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o716 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o717 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o718 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o719 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o720 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o721 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o723 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o725 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o726 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o728 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o729 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o730 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o731 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o732 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o734 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o736 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o737 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o739 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o740 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o742 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o743 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o748 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Command to send: m d o753 e (clientserver.py:501, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:15 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b; CREATE TABLE test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b ENGINE=IcebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/', format=Parquet, url = 'http://minio1:9001/root/') SETTINGS object_storage_cluster = 'cluster_simple' on node1 (cluster.py:3677, query) 2025-04-04 18:36:15 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b on node1 (cluster.py:3677, query) 2025-04-04 18:36:15 [ 633 ] DEBUG : Executing query SELECT * FROM remote('node2',icebergS3Cluster('cluster_simple', s3, filename = 'iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/', format=Parquet, url = 'http://minio1:9001/root/')) on node1 (cluster.py:3677, query) 2025-04-04 18:36:16 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS `test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b` SYNC on node1 (cluster.py:3677, query) 2025-04-04 18:36:16 [ 633 ] DEBUG : Executing query DROP TABLE IF EXISTS test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b; CREATE TABLE test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b ENGINE=IcebergS3(s3, filename = 'iceberg_data/default/test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b/', format=Parquet, url = 'http://minio1:9001/root/') on node1 (cluster.py:3677, query) 2025-04-04 18:36:16 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b on node1 (cluster.py:3677, query) 2025-04-04 18:36:16 [ 633 ] DEBUG : Executing query SELECT * FROM test_iceberg_cluster_2_s3_96923756_3736_4991_a48a_9cc6a591039b SETTINGS object_storage_cluster='cluster_simple' on node1 (cluster.py:3677, query) ---------------------------- Captured log teardown ----------------------------- 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o678 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o655 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o700 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o707 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o709 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o708 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o695 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o744 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o724 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o738 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o745 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o746 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o747 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o749 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o750 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o751 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o752 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o754 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o755 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o756 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command to send: m d o757 e (clientserver.py:501, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2025-04-04 18:36:16 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --project-name rootteststorageiceberg --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/docker-compose.yml stop --timeout 20] (cluster.py:122, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/.env --project-name rootteststorageiceberg --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_hdfs.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_storage_iceberg/_instances-2/node3/docker-compose.yml down --volumes] (cluster.py:122, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node3-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-node2-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-resolver-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-hdfs1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-azurite1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-minio1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Stopping (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Stopped (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy2-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Container rootteststorageiceberg-proxy1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Volume rootteststorageiceberg_data1-1 Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Network rootteststorageiceberg_default Removing (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Volume rootteststorageiceberg_data1-1 Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Stderr: Network rootteststorageiceberg_default Removed (cluster.py:148, run_and_check) 2025-04-04 18:36:37 [ 633 ] DEBUG : Cleanup called (cluster.py:894, cleanup) 2025-04-04 18:36:38 [ 633 ] DEBUG : Docker networks for project rootteststorageiceberg are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-04 18:36:38 [ 633 ] DEBUG : Docker containers for project rootteststorageiceberg are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-04 18:36:38 [ 633 ] DEBUG : Docker volumes for project rootteststorageiceberg are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-04 18:36:38 [ 633 ] DEBUG : Command:[docker container list --all --filter name='^/rootteststorageiceberg-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check) 2025-04-04 18:36:38 [ 633 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup) 2025-04-04 18:36:38 [ 633 ] DEBUG : No running containers for project: rootteststorageiceberg (cluster.py:922, cleanup) 2025-04-04 18:36:38 [ 633 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup) 2025-04-04 18:36:38 [ 633 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup) 2025-04-04 18:36:38 [ 633 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check) 2025-04-04 18:36:38 [ 633 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-04 18:36:38 [ 633 ] DEBUG : Images pruned (cluster.py:947, cleanup) 2025-04-04 18:36:38 [ 633 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup) 2025-04-04 18:36:38 [ 633 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check) 2025-04-04 18:36:38 [ 633 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-04 18:36:38 [ 633 ] DEBUG : Volumes pruned: 1 (cluster.py:958, cleanup) ============================== slowest durations =============================== 123.61s setup test_storage_iceberg/test.py::test_cluster_table_function[azure-1] 64.14s setup test_storage_azure_blob_storage/test_cluster.py::test_select_all 23.95s teardown test_storage_azure_blob_storage/test_cluster.py::test_select_all 21.67s teardown test_storage_iceberg/test.py::test_cluster_table_function[s3-2] 8.06s call test_storage_iceberg/test.py::test_cluster_table_function[azure-1] 3.75s call test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2] 3.67s call test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1] 3.00s call test_storage_iceberg/test.py::test_cluster_table_function[azure-2] 2.40s call test_storage_iceberg/test.py::test_cluster_table_function[s3-1] 2.29s call test_storage_iceberg/test.py::test_cluster_table_function[s3-2] 0.78s call test_storage_azure_blob_storage/test_cluster.py::test_select_all 0.00s teardown test_storage_iceberg/test.py::test_cluster_table_function[azure-1] 0.00s teardown test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2] 0.00s teardown test_storage_iceberg/test.py::test_cluster_table_function[azure-2] 0.00s teardown test_storage_iceberg/test.py::test_cluster_table_function[s3-1] 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[azure-2] 0.00s teardown test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1] 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1] 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[s3-1] 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[s3-2] 0.00s setup test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2] =========================== short test summary info ============================ FAILED test_storage_azure_blob_storage/test_cluster.py::test_select_all - AssertionError: assert 1 a\n2 b == 1 a\n2 b\n1 a\n2 b\n1 a\n2 b FAILED test_storage_iceberg/test.py::test_cluster_table_function[azure-1] - A... FAILED test_storage_iceberg/test.py::test_cluster_table_function[azure-2] - A... FAILED test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1] - As... FAILED test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2] - As... FAILED test_storage_iceberg/test.py::test_cluster_table_function[s3-1] - Asse... FAILED test_storage_iceberg/test.py::test_cluster_table_function[s3-2] - Asse... ======================== 7 failed in 257.79s (0:04:17) ========================= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 528, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_e3k51h --privileged --dns-search='.' --memory=30709026816 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=2 --color=no --durations=0 test_storage_azure_blob_storage/test_cluster.py::test_select_all 'test_storage_iceberg/test.py::test_cluster_table_function[azure-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[azure-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[hdfs-2]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-1]' 'test_storage_iceberg/test.py::test_cluster_table_function[s3-2]' -vvv" altinityinfra/integration-tests-runner:cd6390247eca ' returned non-zero exit status 1.